Parsing a large JSON file for creating wordpress posts

I have a large JSON file (>100MB) which I want to parse and then convert its objects into wordpress posts.
I have successfully created a function to complete this task but it is not able to loop through all object, else it dies by giving an error of PHP Fatal error:
Allowed memory size of 268435456 bytes exhausted (tried to allocate 92 bytes)
Function is able to handle files under 1MB but for files greater than 1MB it fails.
I have asked server administrators to increase memory limit and now no memory errors are visible but still it is not able to fetch all JSON.

I have gone through many post and questions but couldn’t find/understood anything useful. Also it is my first attempt to write such function, so any help/guidance will be appreciated.

Read More

EDIT
Added code for function used

function create_post_from_json($json_key) {

    $json_options = get_option('json_file_data');
    $obj = wp_remote_retrieve_body(wp_remote_get($json_options[$json_key]['url'],array( 'timeout' => -1 )));
    $obj = json_decode($obj);
    $id_stored = array();
    $new_posts_id_array = array();
    $new_json_posts_id_array = get_option('json_' . $json_options[$json_key]['name'] . "_post_ids");
    $id_stored = get_option('json_' . $json_options[$json_key]['name']);
    if(!$id_stored){$id_stored = [];}
    foreach ($obj->products as $one_post) {
        $post_char_id = $one_post->ID;
        $new_posts_id_array[] = $post_char_id;
        $cat_array = array();

        if (!in_array($post_char_id, $id_stored)) {
            $id_stored[] = $post_char_id;
            update_option('json_' . $json_options[$json_key]['name'], $id_stored);
            $post = array(
                'post_title' => $one_post->name,
                'post_status' => 'publish',
                'post_author' => 1,
                'post_content' => $one_post->description,
                'post_type' => 'destinations',
            );

            $new_post_id = wp_insert_post($post); //post id
    return true;
}

Related posts

1 comment

  1. Chances are your script might be running out of time.

    You could increase the max execution time using set_time_limit(100); // 100 seconds

    Or to make it indefinite, use set_time_limit(0);

    Note: Set the time on top of your script.

Comments are closed.