Filtering through tables and removing empty ones

I have several tables from a different database and I need to convert this to something that wordpress will be able to use.

The problem is that those tables are terribly connected. So I have article id’s in the article table, but the article text is in a separate table. And those two are connected with a table that has foreign article_id key in it and a foreign primary key called instance_id. The whole thing looks like this:

Article info is in tco_articles and the text is in tco_module_eostext.

The main problem is that in the tco_article_modules there are ~180000 instance_id keys. Every article_id has 9 different instance_id assigned to it (don’t really know why).

So what I did is:

global $wpdb;
$posts = array();
$byline = array();

$article_id = $wpdb->get_results('SELECT DISTINCT id FROM tco_articles', OBJECT_K);
$instance_id = $wpdb->get_results('SELECT DISTINCT instance_id FROM tco_articles_modules', ARRAY_A);

if I do a print_r($instance_id); I’ll get something like

Array
(
    [0] => Array
        (
            [instance_id] => 928615
        )

    [1] => Array
        (
            [instance_id] => 928616
        )...

This gives ~21000 $article_ids and ~180000 instance_ids. If I go and manually see what I’ll get for one instance_id in the tables related to tco_articles_modules with, for instance:

$wpdb->get_results('SELECT * FROM tco_module_byline WHERE instance_id=123456', ARRAY_A);

Where I’ve put the dummy key 123455 just for illustration. I could get empty array as a result. But If I pick another key, I could have something inside. I have no way of knowing for what instance_id I’ll have a filled row in a certain table.

So my idea was to loop through every instance_id with foreach, and do a search through every table, and if they are not empty I’d get something that I can use. So I tried:

foreach ($instance_id as $key => $value) {
    $single_id = $value['instance_id'];
    $byline_out = $wpdb->get_results( "SELECT * FROM tco_module_byline WHERE instance_id='.$single_id.' IS NOT NULL ", OBJECT_K);
    $byline[] = $byline_out;
}

I get nothing. If I remove IS NOT NULL I either get empty arrays, and one time, I forgot what I put in the sql query, I got a bunch of empty arrays and some that were filled.

I’d like to:

1. Either eliminate the empty ones right along in the query

2. Make an efficient way of querying this whole thing (every table connected to the article) and making an array of posts, that I can then use to insert into wordpress own table with wp_insert_post

I’m not at all expert at sql and queries, but from what I’ve read, if my table is structured properly this shouldn’t be a problem. However this thing is so badly interconnected that I cannot find an easy way of getting out all the data I need. : Also doing a foreach for 180000 keys seems like an awful way of querying data.

So if anyone has any help or advice, I’d appreciate it.

EDIT

So I’ve wanted to limit my foreach to go through only a few keys and I did this:

$i=0;

foreach ($instance_id as $key => $value) {
    $single_id = $value['instance_id'];
    $byline_out = $wpdb->get_results( "SELECT * FROM tco_module_byline WHERE instance_id='.$single_id.' IS NOT NULL", OBJECT_K);
    $byline[] = $byline_out;
    if (++$i == 1) break;
}

I get results with filled values from byline table:

Array
(
    [0] => Array
        (
            [1086742] => stdClass Object
                (
                    [instance_id] => 1086742
                    [show_name] => 0
                    [meta] => 
                )

            [1086789] => stdClass Object
                (
                    [instance_id] => 1086789
                    [show_name] => 0
                    [meta] => 
                )...

I think that my foreach is faulty. But I’m not sure how :S