Ask a Question Register Login

guzzle - How to process (Inserting in db) incoming streaming data in php?

379

I am continuously getting streaming twitter data from gnip by guzzle library in json formate.

Below is my code

    require_once('vendor/autoload.php');
    $url =https://stream.gnip.com:443/accounts/{account_name}/publishers/twitter/streams/track/dev.json";
    $username = '*****';
    $password = '*****';
    $headers = array('Accept-Encoding' => 'gzip');

    $client = new GuzzleHttp\Client();
    $client->setDefaultOption('headers/Accept-Encoding', 'gzip');
    $data = $client->get($url, ['auth' => [$username, $password]]))
    {
      $tdata = $data->getBody();
     //here i am getting json data continuously,that I have to insert in my db.
     //I have written my code to insert in db and it is inserting in db but after some      time it will not insert in db.


    }

So please suggest me ,if there are other best method or some php library available.

226

Answer

Solution:

For best practice, you can implement a rest service for inserting data in db. Let say you have a rest service;

/data/insert

You will send each streamed data to that service like;

functiond insertData($data) {}
    $url = 'http://domain.com/data/insert';
    $fields = array(
                            'field1' => urlencode($data["field1"]),
                            'field2' => urlencode($data["field1"]),
                            'field3' => urlencode($data["field1"])
                    );

    foreach($fields as $key=>$value) { 
        $fields_string .= $key.'='.$value.'&'; 
    }
    rtrim($fields_string, '&');

    $ch = curl_init();

    curl_setopt($ch,CURLOPT_URL, $url);
    curl_setopt($ch,CURLOPT_POST, count($fields));
    curl_setopt($ch,CURLOPT_POSTFIELDS, $fields_string);

    $result = curl_exec($ch);

    curl_close($ch);
}

When you get stream data, just run this function, and do not care about how to insert data in db. Maybe, you can handle data insertion part in service side.

People are also looking for solutions to the problem: php - Query Database with Where Clause depending on array values

Source

Didn't find the answer?

Our community is visited by hundreds of web development professionals every day. Ask your question and get a quick answer for free.

Ask a Question

Write quick answer

Do you know the answer to this question? Write a quick response to it. With your help, we will make our community stronger.

Similar questions

Find the answer in similar questions on our website.