BashScripts.org

It is currently Wed May 22, 2013 6:45 pm

losing data with wait

Board index » The Scripts » Requests

All times are UTC - 6 hours

Page 1 of 1

[ 14 posts ]

Print view

Previous topic | Next topic

Author

Message

japie

Posted: Sun May 31, 2009 11:48 pm

Joined: Sun May 31, 2009 11:29 pm
Posts: 10

Hello,

I am using a script to read values from owfs sensors and put them into mysql, only when I add function to simultaniously read all entrys the data seems to get lost, example:

Code:

   time_step=$(sleep 1) &
   time_step_pid=$!
   temp_cyl_1=$(echo "scale=0;$(cat /tmp/owfs/30.03DD95110000/typeK/temperature)/1" | bc) &
   temp_cyl_1_pid=$!
   temp_cyl_2=$(echo "scale=0;$(cat /tmp/owfs/30.5FDF95110000/typeK/temperature)/1" | bc) &
   temp_cyl_2_pid=$!
   wait $time_step_pid $temp_cyl_1_pid $temp_cyl_2_pid
   values=$temp_cyl_1,$temp_cyl_2
   CMD="mysql -u root dlogger -e \"insert into samples values ($values)\""
   `eval $CMD`

If I replace the eval with an echo I get:
mysql -u root dlogger -e "insert into samples values (,)"
If I get rid of the pid and wait I get the data, does anyone knows what I am doing wrong?

p.s. owfs is like /proc, the contens of the temperature files is something like 278.4

Top

fredrik.eriksson

Posted: Mon Jun 01, 2009 2:38 am

Joined: Mon Nov 17, 2008 7:25 am
Posts: 221

Why not just let the bc do it's calculations in due time?
Is there some particular reason to wait for the process to finish?

Unless you're trying to thread this that method is not worth using.

If you need to wait for the process there is not obvious reason to let it run in the background.

As so many other have said... K.I.S.S (keep it simple, stupid)

Best regards
Fredrik Eriksson

Top

japie

Posted: Mon Jun 01, 2009 3:00 am

Joined: Sun May 31, 2009 11:29 pm
Posts: 10

The script is used to log 1-wire sensor data to mysql on our tractor-puller http://www.minimumrisk.nl
The mysql data is available true webinterface http://www.dlogger.deserver.nl

The main issue is the rpm reading, this is a counter connected to the crankshaft so I subtract the previous reading from the current one and then divide the difference with the timestep.
Unfortunatly reading the sensors isn't as fast as /proc so reading them one at a time takes 0.6sec. to 1.1sec. so I am trying to read them all at once at a fixed timebase so I can devide the rpm result by the fixed timeframe.
That is why $time_step is part of the wait line.

If you like I can post the full script?

I know all of it is a bit much for a script but for me it was the most conveniant way to do it, working on a lathe is easyer for me so if you have any suggestions or help you have my ethernal thanks...

Top

fredrik.eriksson

Posted: Mon Jun 01, 2009 3:21 am

Joined: Mon Nov 17, 2008 7:25 am
Posts: 221

Okey, but then i see your point.

I would actually do this in 2 scripts instead then.

One that reads /proc constantly, meaning just

Code:

#!/bin/bash
file_path=/where/to/put/the/files

function get_filename {
   x=$2
   seq_file=$(printf "%04u" $x) 
   filename="$file_path/probe_$1_$seq_file"
   return $filename
}

x=0
while [ true ]; do 
   content[2]=$(cat /proc/file/to/read)
   content[3]=$(cat /proc/second/file/to/read)
   if [ "${content[0]}" != "${content[2]}" ]; then
      content[0] = ${content[2]}
      content[1] = ${content[3]}
      filename=$(get_filename 1 $x)
      ((++x)
      echo "$content[0]" >> $filename
      filename=$(get_filename 2 $x)
      ((++x)
      echo "$content[1]" >> $filename
   fi
done

This should create a file named "probe0001" which contains a value from your proc file. It will probe it all the time as soon as it can and should only create a new file when the value has changed.

The second script is something i would do like this:

Code:

#!/bin/bash

while [ true ]; do
   file_1=$(ls --color=no /path/to/probe/files/probe_1* | head -n1)
   file_2=$(ls --color=no /path/to/probe/files/probe_2* | head -n1)
   content_1=$(cat $file_1)
   content_2=$(cat $file_2)
   Do what ever you want with content_1 and content_2

   sleep 1
done

Hope this is a better way of doing it

It's atleast what i would've done.

Best regards
Fredrik Eriksson

Top

japie

Posted: Mon Jun 01, 2009 4:38 am

Joined: Sun May 31, 2009 11:29 pm
Posts: 10

fredrik.eriksson wrote:

This should create a file named "probe0001" which contains a value from your proc file. It will probe it all the time as soon as it can and should only create a new file when the value has changed.

That is fresh, but the first thing that comes in my mind why copy the contens of a file to another file.
An advantage of this way is that the step times of reading and processing are seperate.
owfs has some way for this too, it has an uncached directory and the cached (normal) dir is refreshed whenever the uncached is accessed, so a read every XXXms would also refresh the normal reading.

fredrik.eriksson wrote:

The second script is something i would do like this:

Code:

#!/bin/bash

while [ true ]; do
   file_1=$(ls --color=no /path/to/probe/files/probe_1* | head -n1)
   file_2=$(ls --color=no /path/to/probe/files/probe_2* | head -n1)
   content_1=$(cat $file_1)
   content_2=$(cat $file_2)
   Do what ever you want with content_1 and content_2

   sleep 1
done

Hope this is a better way of doing it

It's atleast what i would've done.

Best regards
Fredrik Eriksson

I have tried a while to, but the problem is the processing lines beneeth it, there have to be done some calculations for every sensor and all of them together is giving some "lag" and therefore making the rpm reading/calculation unreliable.
That was the main reason for using pid and wait.
At the moment I have almost the same running without the wait function and it does 47 readings per minute but that means the rpm calculation is not reliable so I need to grab it all in a fixed timeframe.

Here the full script as I have so far:

Code:

#!/bin/sh

LANG=C
LC_ALL=C
export LANG LC_ALL

counter_1=$(cat /tmp/owfs/1D.E4D90D000000/counters.A)

read_owfs ()
{
   time_step=$(sleep 1) &
   time_step_pid=$!
   time_stamp=$(date +%y%m%d%H%M%S) &
   time_stamp_pid=$!   
   rpm_1=$(cat /tmp/owfs/1D.E4D90D000000/counters.A)-$counter_1
   rpm_eng=$(echo "scale=0;($rpm_1/1)*60/2" | bc)
   counter_1=$(cat /tmp/owfs/1D.E4D90D000000/counters.A) &
   rpm_eng_pid=$!
   pres_trb_air=$(cat /tmp/owfs/26.61D1BC000000/VAD)
   pres_trb_air=$(echo "scale=2;(-0.5*($pres_trb_air*$pres_trb_air)+0.2*$pres_trb_air+7)*100" | bc) &
   pres_trb_air_pid=$!
   pres_eng_oil=$(cat /tmp/owfs/26.61CDBC000000/VAD)
   pres_eng_oil=$(echo "scale=2;(0.84*($pres_eng_oil*$pres_eng_oil)+-0.2*$pres_eng_oil+0.5)*100" | bc) &
   pres_eng_oil_pid=$!
   temp_cyl_1=$(echo "scale=0;$(cat /tmp/owfs/30.03DD95110000/typeK/temperature)/1" | bc) &
   temp_cyl_1_pid=$!
   temp_cyl_2=$(echo "scale=0;$(cat /tmp/owfs/30.5FDF95110000/typeK/temperature)/1" | bc) &
   temp_cyl_2_pid=$!
   temp_cyl_3=$(echo "scale=0;$(cat /tmp/owfs/30.4CE495110000/typeK/temperature)/1" | bc) &
   temp_cyl_3_pid=$!
   temp_cyl_4=$(echo "scale=0;$(cat /tmp/owfs/30.08DD95110000/typeK/temperature)/1" | bc) &
   temp_cyl_4_pid=$!
   temp_trb_air=$(echo "scale=0;$(cat /tmp/owfs/30.ADDD95110000/typeK/temperature)/1" | bc) &
   temp_trb_air_pid=$!
   temp_eng_oil="0" &
   temp_eng_oil_pid=$!
   wait $time_step_pid $time_stamp_pid $temp_cyl_1_pid $temp_cyl_2_pid $temp_cyl_3_pid $temp_cyl_4_pid $temp_trb_air_pid $pres_trb_air_pid $temp_eng_oil_pid $pres_eng_oil_pid $rpm_eng_pid
   values=$time_stamp,$temp_cyl_1,$temp_cyl_2,$temp_cyl_3,$temp_cyl_4,$temp_trb_air,$pres_trb_air,$temp_eng_oil,$pres_eng_oil,$rpm_eng
   CMD="mysql -u root dlogger -e \"insert into samples values ($values)\""
#   `eval $CMD`
   echo $CMD
}

while [ 1 ]
do
   read_owfs
done

exit 0

Top

fredrik.eriksson

Posted: Mon Jun 01, 2009 7:05 am

Joined: Mon Nov 17, 2008 7:25 am
Posts: 221

Maybe a better datamining language would be better for this? Bash still is just a shell interpreter with some nice script features, but still relies heavily on executing external programs for every part of the nice features.

Might i suggest perl or php?
The only way i can think of doing this faster is to store the value given from your proc files directly instead of doing the calculations in your script.

If you just insert the value and then retrieve when you want to figure out the rpm then you probably would save time?

Best regards
Fredrik Eriksson

Top

japie

Posted: Mon Jun 01, 2009 9:20 am

Joined: Sun May 31, 2009 11:29 pm
Posts: 10

fredrik.eriksson wrote:

Might i suggest perl or php?

I know Fredrik,

I started in bash since that is the best I can do (don't start yelling: "Is your perl/php even worse

but I started recently with a rewrite in php, php has bbuiltin functions for mysql and a library for owfs but I have up to Saterday to make things work with bash because of the pulling seison starting, next winter I will start the rewrite...
I tought I had a brilliant idea with the wait function but sadly it seems to loose the contens of the variables somewhere and I don't understand why.

At the moment it is showing me the wanted data, I yust came back from our testrun this afternoon (data online all-ready) and the most important stuff (cylinder temps.) are logged fine, rpm is way of and stuff could move quiker but it works. (for now)

Thanks for helping so far!

Top

fredrik.eriksson

Posted: Mon Jun 01, 2009 9:56 am

Joined: Mon Nov 17, 2008 7:25 am
Posts: 221

since i'm apperantly abit faster in the php part i've written this... you might have to modify it abit thou.

Code:

<?
   function start_mysql($user, $password, $db) {
      mysql_connect("localhost",$user,$password) or die(mysql_error());
      mysql_select_db($db) or die(mysql_error());
   }

   function read_owfs ($counter) {
      $now = date("ydmHis");

      $rpm = file_get_contents("/tmp/owfs/1D.E4D90D000000/counters.A") - $counter;

      $pres_trb_air = file_get_contents("/tmp/owfs/26.61D1BC000000/VAD");
      $pres_trb_air = (-0.5*($pres_trb_air*$pres_trb_air)+0.2*$pres_trb_air+7)*100;

      $pres_eng_oil = file_get_contents("/tmp/owfs/26.61CDBC000000/VAD");
      $pres_eng_oil = (-0.5*($pres_trb_air*$pres_trb_air)+0.2*$pres_trb_air+7)*100;

      $temp_cyl[] = file_get_contents("/tmp/owfs/30.03DD95110000/typeK/temperature") / 1;
      $temp_cyl[] = file_get_contents("/tmp/owfs/30.5FDF95110000/typeK/temperature") / 1;
      $temp_cyl[] = file_get_contents("/tmp/owfs/30.4CE495110000/typeK/temperature") / 1;
      $temp_cyl[] = file_get_contents("/tmp/owfs/30.08DD95110000/typeK/temperature") / 1;
      $temp_trb_air = file_get_contents("/tmp/owfs/30.ADDD95110000/typeK/temperature)/1") / 1;
      $temp_eng_oil = 0;

      $values = "$now";
      foreach($temp_cyl as $key) {
         $values .= ",$key";
      }
      $values .= ",$temp_trb_air,$pres_trb_air,$temp_eng_oil,$pres_eng_oil,$rpm_eng";

      $q = "insert into samples values ($values)";
      mysql_query($q);
   }

   $username = "root";
   $password = "password";
   $database = "dlogger";

   start_mysql($username, $password, $database);

   while(true) {
      $counter = file_get_contents("/tmp/owfs/1D.E4D90D000000/counters.A");
      read_owfs($counter);
      sleep(1);
   }
?>

This does not wait for anything, it sleeps for 1 second and then runs again until you ctrl+c it..
You can use usleep() if you want to lower the sleep timer. But remember that it uses microseconds, so 250000 is 1/4 second.

Also added a function to connect to a mysql, just change the username, password and database variables to the right ones

The now variable will contain YearMonthDayHourMinuteSeconds

Hope this gives you nudge in the right direction

Best regards
Fredrik Eriksson

Top

japie

Posted: Fri Jun 05, 2009 12:41 pm

Joined: Sun May 31, 2009 11:29 pm
Posts: 10

Thanks Fredrik,

But since you probebly asumed since I didn't respond very fast it I have other prioritys at the moment, the engine is running excelent, but there are a few items I have to solve before May 13 (first real pull), oil pressure is only 200kpa so have to place a new pump and the exhaust temps are over 900 degrees celcius so I need water injection fast...

Watch the minimumrisk site for news and if you want more info or refresh contact (if I forget in the hectic) mail me via minimumrisk...

Thanks for helping, Japie.

Top

japie

Posted: Fri Jun 26, 2009 11:20 pm

Joined: Sun May 31, 2009 11:29 pm
Posts: 10

Finaly got some time to try the php script but it suffers the same flaw as my bash script.
Reading a sensor takes between 0.1 and 0.5 sec. so for a reading in a fixed timestep the script must not wait until the sensor delivers data.
Is there someting like

Code:

in php?

Top

fredrik.eriksson

Posted: Sat Jun 27, 2009 4:47 am

Joined: Mon Nov 17, 2008 7:25 am
Posts: 221

you could always build it with threading, but that's a tricky business.
What I would recommend is that you start using a mysql db and run a separate process to handle the reading of data from your files.
And a "reporting" process to create the values from your database.

I can't think of many other ways of doing this. "threading" is the way to go I would say.

This is a bit of threading code, it creates a child which reads each of the counter files and inserts it into the mysql database.
This might be what you are looking for, but keep in mind I haven't tested this _at all_ so there might be some bugs and features

Code:

<?
   function sig_handler($signo) {
      switch($signo) {
         case SIGTERM:
            exit;
            break;
         case SIGINT:
            exit;
            break;
         default:
            break;
      }
      return false;
   }


   function start_mysql($user, $password, $db) {
      mysql_connect("localhost",$user,$password) or die(mysql_error());
      mysql_select_db($db) or die(mysql_error());
   }

   function read_owfs ($counter) {
      $now = date("ydmHis");

      $rpm = file_get_contents("/tmp/owfs/1D.E4D90D000000/counters.A") - $counter;

      $pres_trb_air = file_get_contents("/tmp/owfs/26.61D1BC000000/VAD");
      $pres_trb_air = (-0.5*($pres_trb_air*$pres_trb_air)+0.2*$pres_trb_air+7)*100;

      $pres_eng_oil = file_get_contents("/tmp/owfs/26.61CDBC000000/VAD");
      $pres_eng_oil = (-0.5*($pres_trb_air*$pres_trb_air)+0.2*$pres_trb_air+7)*100;

      $temp_cyl[] = file_get_contents("/tmp/owfs/30.03DD95110000/typeK/temperature") / 1;
      $temp_cyl[] = file_get_contents("/tmp/owfs/30.5FDF95110000/typeK/temperature") / 1;
      $temp_cyl[] = file_get_contents("/tmp/owfs/30.4CE495110000/typeK/temperature") / 1;
      $temp_cyl[] = file_get_contents("/tmp/owfs/30.08DD95110000/typeK/temperature") / 1;
      $temp_trb_air = file_get_contents("/tmp/owfs/30.ADDD95110000/typeK/temperature)/1") / 1;
      $temp_eng_oil = 0;

      $values = "$now";
      foreach($temp_cyl as $key) {
         $values .= ",$key";
      }
      $values .= ",$temp_trb_air,$pres_trb_air,$temp_eng_oil,$pres_eng_oil,$rpm_eng";

      $q = "insert into samples values ($values)";
      mysql_query($q);
   }

   $username = "root";
   $password = "password";
   $database = "dlogger";

   start_mysql($username, $password, $database);

   // Child pid array.
   $pid = array();
   // This array holds all the counters.
   $counters = array(
                     "/tmp/owfs/1D.E4D90D000000/counters.A",
                     "/tmp/owfs/1D.E4D90D000000/counters.A",
                     //"Add more here",
                     );

   // This will spawn children for each of the counters in $counters array.
   for($i = 0; $i < count($counters); $i++) {
      // This forks the process to create a child.
      $pid[$i] = pcntl_fork();
      // Make sure all our kids handle ctrl+c and ctrl+break properly.
      // If you need it to handle some other SIG handler you can add them in the function sig_handler()
      pcntl_signal(SIGTERM, "sig_handler");

      if(!$pid[$i]) {
         while(true) {
            read_owfs($counters[$i]);
            // Here you can put other things that is supposed to be done with this counter.
         }
         exit;
      }
   }

   foreach($pid as $key) {
      pcntl_waitpid($key, $status, WUNTRACED);
   }

   exit;
?>

Top

japie

Posted: Fri Jul 10, 2009 11:16 am

Joined: Sun May 31, 2009 11:29 pm
Posts: 10

Got it! (in bash)

Main problem with the & is that every variable goes into oblivian, so instead of defining a variable I write them to files in /tmp and a second script that reads those files and writes the entrys into mysql.
Working very very well...
Real life test next Sunday, next match...

Top

japie

Posted: Fri Jul 10, 2009 11:20 am

Joined: Sun May 31, 2009 11:29 pm
Posts: 10

Oh Fredric sorry forgot, the php code is still in my work folder, going to rewrite all the bash stuff to php but not right now, isn't there a simpler way to do the following in php? (I found your previous post hard to understand due to the counter mixed in)

Code:

#!/bin/sh

LANG=C
LC_ALL=C
export LANG LC_ALL

mkdir /tmp/sensors
counter_A_old=$(cat /tmp/owfs/uncached/1D.E4D90D000000/counters.A)
echo $counter_A_old > /tmp/sensors/counter_A_old

rpm_eng ()
{    
   counter_A_old=$(cat /tmp/sensors/counter_A_old)
   counter_A_new=$(cat /tmp/owfs/uncached/1D.E4D90D000000/counters.A)
   rpm_eng=$(echo "scale=0;($counter_A_new-$counter_A_old)*30/2" | bc)
   echo $counter_A_new > /tmp/sensors/counter_A_old
   echo $rpm_eng > /tmp/sensors/rpm_eng
}

temp_cyl_1 ()
{  
   temp_cyl_1=$(echo "scale=0;$(cat /tmp/owfs/uncached/30.03DD95110000/typeK/temperature)/1" | bc)
   echo $temp_cyl_1 > /tmp/sensors/temp_cyl_1
}   

temp_cyl_2 ()
{
   temp_cyl_2=$(echo "scale=0;$(cat /tmp/owfs/uncached/30.5FDF95110000/typeK/temperature)/1" | bc)
   echo $temp_cyl_2 > /tmp/sensors/temp_cyl_2
}

temp_cyl_3 ()
{   
   temp_cyl_3=$(echo "scale=0;$(cat /tmp/owfs/uncached/30.4CE495110000/typeK/temperature)/1" | bc)
   echo $temp_cyl_3 > /tmp/sensors/temp_cyl_3
}

temp_cyl_4 ()
{   
   temp_cyl_4=$(echo "scale=0;$(cat /tmp/owfs/uncached/30.08DD95110000/typeK/temperature)/1" | bc)
   echo $temp_cyl_4 > /tmp/sensors/temp_cyl_4
}

temp_trb_air ()
{   
   temp_trb_air=$(echo "scale=0;$(cat /tmp/owfs/uncached/30.ADDD95110000/typeK/temperature)/1" | bc)
   echo $temp_trb_air > /tmp/sensors/temp_trb_air
}

pres_trb_air ()
{   
   pres_trb_air=$(cat /tmp/owfs/uncached/26.61D1BC000000/VAD)
   pres_trb_air=$(echo "scale=0;(-0.6*($pres_trb_air*$pres_trb_air)+0.1*$pres_trb_air+10)*100" | bc)
   echo $pres_trb_air > /tmp/sensors/pres_trb_air
}

pres_trb_exh ()
{   
   pres_trb_exh=$(cat /tmp/owfs/uncached/26.64A080000000/VAD)
   pres_trb_exh=$(echo "scale=0;(-0.6*($pres_trb_exh*$pres_trb_exh)+-0.1*$pres_trb_exh+10)*100" | bc)
   echo $pres_trb_exh > /tmp/sensors/pres_trb_exh
}

pres_eng_oil ()
{   
   pres_eng_oil=$(cat /tmp/owfs/uncached/26.61CDBC000000/VAD)
   pres_eng_oil=$(echo "scale=0;(0.84*($pres_eng_oil*$pres_eng_oil)+-0.2*$pres_eng_oil+0.5)*100" | bc)
   echo $pres_eng_oil > /tmp/sensors/pres_eng_oil
}

temp_eng_oil ()
{   
   temp_eng_oil=$(echo "scale=0;$(cat /tmp/owfs/uncached/30.50E795110000/typeK/temperature)/1" | bc)
   echo $temp_eng_oil > /tmp/sensors/temp_eng_oil
}
   
read_owfs ()
{
   time_step=$(usleep 500000) &
   time_step_pid=$!
   rpm_eng & 
   rpm_eng_pid=$!
   temp_cyl_1 &
   temp_cyl_1_pid=$!
   temp_cyl_2 &
   temp_cyl_2_pid=$!
   temp_cyl_3 &
   temp_cyl_3_pid=$!
   temp_cyl_4 &
   temp_cyl_4_pid=$!
   temp_trb_air &
   temp_trb_air_pid=$!
   pres_trb_air &
   pres_trb_air_pid=$!
   pres_trb_exh &
   pres_trb_exh_pid=$!
   pres_eng_oil &
   pres_eng_oil_pid=$!
   temp_eng_oil &
   temp_eng_oil_pid=$!
   wait $time_step_pid $rpm_eng_pid $temp_cyl_1_pid $temp_cyl_2_pid $temp_cyl_3_pid $temp_cyl_4_pid $temp_trb_air_pid $pres_trb_air_pid $pres_trb_exh_pid $temp_eng_oil_pid $pres_eng_oil_pid
}

while [ 1 ]
do
   read_owfs
done

exit 0

Top

fredrik.eriksson

Posted: Sat Jul 11, 2009 9:26 am

Joined: Mon Nov 17, 2008 7:25 am
Posts: 221

Oh sorry, I thought I explained it.
The counters thingie is just an array containing the filenames for your counters.

I've done that just so it can spawn a new thread per counter-file.

Thing is you could just as well do something with a for loop or something that just spawns X threads that reads the same

this way was just easier.

Threads is something of a mystery and they're usually quite difficult to understand.
When invoking fork_pcntl() you create another process that starts the same script from that position (from where it was forked).
In an example where you only fork the process 1 time you get 1 parent and 1 child where the parent is the scripts original executioner.
The child will get the child's PID returned where as the parent will only get 0 returned from the function, this way you can distinguish them from each other.

In this case I fork it X times (depending on the amount of array elements in $counters). Just to not get any wait lagg (for each counter) each process will insert data to the database on their own and doesn't depend on the other counters to deliver their data in time.

Best regards
Fredrik Eriksson

Top

Page 1 of 1

[ 14 posts ]

Board index » The Scripts » Requests

All times are UTC - 6 hours

Who is online

Users browsing this forum: No registered users and 9 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

BashScripts | Promote Your Page Too

losing data with wait

Who is online