How to break down importing of feeds

3 comments

1. You can try using wp cron functions, programing a feed import every 10 minutes or so, until you have nothing left in queue.

   Example code:

   <?php
   register_activation_hook( __FILE__, 'wp1903_schedule_cron' );
   
   function wp1903_schedule_cron() {
       $counter = 0;
       $feeds = array(
           'http://example.com/feed1',
           'http://example.com/feed2',
           'http://example.com/feed3',
           'http://example.com/feed4'
       );
   
       foreach ($feeds as $feed) {
           $counter++;
           // schedule every 10 mins
           $time = time() + (600 * $counter);
           wp_schedule_single_event( $time, 'wp1903_process_feed', $feed );
       }
   }
   
   add_action('wp1903_process_feed', 'wp1903_import_feed');
   
   function wp1903_import_feed($feed_url = '') {
       // do the import
   }
   

   It is essential only to run the scheduler on a activation hook, or else at every page load the crons will be rescheduled.

   Log in to Reply

2. You’ve tried to extend the script’s maximum execution time with set_time_limit()? This will not cause the server to crash (because it has a memory limit too):

   set_time_limit( 0 ); // No time limit is imposed
   

   I’m not sure what you mean with “breaking down” the script, you might want to have a second thought on that.

   What you could do if there are to many rows in the feed’s XML you also could make your own import script and take care of its handle your own way by processing small part by small part.

   $xml_handler = new ProductsParser();
   $xml_parser  = xml_parser_create();
   
   $source = '--URL--';
   
   xml_set_object( $xml_parser, $xml_handler );
   xml_parser_set_option( $xml_parser, XML_OPTION_CASE_FOLDING, false );
   xml_set_element_handler( $xml_parser, 'startElement', 'endElement' );
   xml_set_character_data_handler( $xml_parser, 'cdata' );
   
   $fp = fopen ( $source, 'r' );
   while ( $chunk = fread( $fp, 4096 ) ) {
     xml_parse( $xml_parser, $chunk, feof( $fp ) );
     flush();
   }
   
   fclose( $fp );
   
   class ProductsParser {
     public $product; # Holds the record values
     public $elem_key; # Holds the current element key while reading
     public $record; # Holds the record tag
   
     function __construct( $args ) {
       $this->product  = false;
       $this->elem_key = false;
       $this->record   = '--RECORD-NAME--';
     }
   
     function startElement( $parser, $tag, $attributes ) {
       if ( $this->record == $tag && ! is_array( $this->product ) ) {
         $this->product = array();
       } elseif( is_array( $this->product ) ) {
         $this->elem_key = $tag;
       }
     }
   
     function endElement( $parser, $tag ) {
       if ( $this->record == $tag && is_array( $this->product ) ) {
         // Process the product row
   
         $this->product = false;
       } elseif ( is_array( $this->product ) && $this->elem_key != false ) {
         $this->elem_key = false;
       }
     }
   
     function cdata( $parser, $cdata ) {
       if ( is_array( $this->product ) && $this->elem_key != false ) {
         $this->product[$this->elem_key] = $cdata;
       }
     }
   }
   

   More information about the XML parser can be found in the PHP manual.

   Log in to Reply

3. When it comes down to high execution time scripts I usually split it up as simple as possible using ajax so here is a concept class that should get you started with creating an admin page and ajaxed import loop:

   <?php
   class Ajax_Import{
   
       public function __construct(){
           if(is_admin()){
               add_action('admin_menu', array($this, 'add_plugin_page'));
           }
           add_action('wp_ajax_do_feed_import',array($this,'do_import'));
       }
   
       public function add_plugin_page(){
           // This page will be under "Settings"
           add_options_page('ajaxed Import', 'ajaxed Import', 'manage_options', 'test-setting-admin', array($this, 'create_admin_page'));
       }
   
       public function create_admin_page(){
           wp_enqueue_script( 'jquery');
           ?>
           <div class="wrap">
               <?php screen_icon(); ?>
               <h2><?php _e('AJAXed Import'); ?></h2>          
               <table class="form-table">
                   <tbody>
                       <th scope="row">
                           <label for="my-text-field"><?php _e('Feeds To Import'); ?></label>
                       </th>
                       <td>
                           <textarea id="feeds"></textarea>
                           <br>
                           <span class="description"><?php _e('Enter Each feed in a new line.'); ?></span>
                       </td>
                   </tbody>
               </table>
               <p class="submit"><input type="submit" id="do_import" value="Save Changes" class="button-primary" name="Submit"></p>
               <div id="import_status" style="display:none"></div>
           </div>
           <?php
           $this->import_loop_js();
       }
   
       public function do_import(){
           //do your import here ex:
           $rss = fetch_feed(esc_url( $_POST['feed_url']));
           if (!is_wp_error( $rss ) ) {
               $count = $rss->get_item_quantity(); 
               //do actuall import loop or whatever
               echo json_encode(array('message' => '<div id="message" class="success"><p>imported ' . $count . ' items from '.esc_url( $_POST['feed_url']).'</p></div>'));
           }else{
               echo json_encode(array('errors' => '<div id="message" class="error"><p>' . $rss->get_error_message() . '</p></div>'));
           }
           die();
       }
   
       public function import_loop_js(){
           ?>
           <script type="text/javascript">
           jQuery(document).ready(function($){
               var ajax_import = (function(){
                   var instantiated;
                   var feeds;
                   function init (){
                       return {
                           start_import: function(){
   
                               instantiated.feeds = $("#feeds").val().split('n');
                               instantiated.update_status('', true);
                               instantiated.next_feed();
                           },
                           make_ajax_call:function(feed){
                               data = {
                                   action: "do_feed_import",
                                   feed_url: feed
                               };
                               $.post( 
                                   ajaxurl,
                                   data, 
                                   function(res){
                                       if (res.error){//add error message
                                           instantiated.update_status(res.error);
                                       }else{//add success message
                                           instantiated.update_status(res.message);
                                       }
                                       //call next feed
                                       instantiated.next_feed();
                                   }, 
                                   'json'
                               );
                           },
                           next_feed:function(){
                               if (instantiated.feeds.length > 0) {//if we have feeds in the list starts ajax calls
                                   instantiated.make_ajax_call(instantiated.feeds.shift());
                               } else{
                                   instantiated.update_status("all Done!!!");
                                   $("#do_import").removeAttr("disabled");
                               }
                           },
                           update_status:function(m,clear){
                               clear = clear || false;
                               if (clear)
                                   $("#import_status").html('').show();
                               else
                                   $("#import_status").append(m);
                           }
                       }
                   }
   
                   return {
                       getInstance :function(){
                           if (!instantiated){
                               instantiated = init();
                           }
                           return instantiated; 
                       }
                   }
               })();
               $("#do_import").on('click',function(){
                   $(this).attr('disabled','disabled');
                   ajax_import.getInstance().start_import();
               });
           });
           </script>
           <?php
   
       }
   }
   
   $Ajax_Import = new Ajax_Import();
   

   Log in to Reply