Problems with Bash read on unslung

[Brian-253]

Hi, using unslung Firmware Version:  V2.3R63-uNSLUng-6.8-beta and
trying to port a bash script from a Debian slug. I installed the
newest Bash for unslung. The following does not work:

#!/bin/bash
set -x
rtime=`date +%H:%M`

STRING3="#Jul  4 15:35:06 Slug2 sshd[2501]: Failed password for root
from 192.168.123.165 port 4701 ssh2"

{ read "A1" "A2" "A3" "A4" "A5" "A6" "A7" "A8" "A9" "A10" "A11" "A12"
"A13" "A14" "A15"; } < <( echo "$STRING3" )

The read command gives the following error:
./mailtext2.sh: line 19: /dev/fd/62: No such file or directory

This does work on Debian bash.
Anyone know how to get this to work?

If not, how else can I parse a single line of text passed to the
script as $1, which has multiple words, into separate words?

Just FYI,

I understand that "< <( echo "$STRING3" )" works like this:

<( echo "$STRING3" ) makes the data look like its coming from a file

< <( echo "$STRING3" ) and the extra < now redirects that data to the
read command.


Cheers Brian

[Marcel Nijenhof]

On Tue, 2008-07-15 at 14:00 +0000, bloedmann999 wrote:

> #!/bin/bash
> set -x
> rtime=`date +%H:%M`
>
> STRING3="#Jul  4 15:35:06 Slug2 sshd[2501]: Failed password for root
> from 192.168.123.165 port 4701 ssh2"
>
> { read "A1" "A2" "A3" "A4" "A5" "A6" "A7" "A8" "A9" "A10" "A11" "A12"
> "A13" "A14" "A15"; } < <( echo "$STRING3" )
>

set -- ${STRING3}

A1=$1
A2=$2
...

--
Marcel Nijenhof
Optware package developer
nslu2@...

[Brian-253]

--- In nslu2-linux@..., Marcel Nijenhof <nslu2@...> wrote:

>
> On Tue, 2008-07-15 at 14:00 +0000, bloedmann999 wrote:
>
> > #!/bin/bash
> > set -x
> > rtime=`date +%H:%M`
> >
> > STRING3="#Jul  4 15:35:06 Slug2 sshd[2501]: Failed password for root
> > from 192.168.123.165 port 4701 ssh2"
> >
> > { read "A1" "A2" "A3" "A4" "A5" "A6" "A7" "A8" "A9" "A10" "A11" "A12"
> > "A13" "A14" "A15"; } < <( echo "$STRING3" )
> >
>
> set -- ${STRING3}
>
> A1=$1
> A2=$2
> ...
>
> --
> Marcel Nijenhof
> Optware package developer
> nslu2@...
>

Marcel,

Ah, seen that somewhere before too. I have one problem with that that
I am investigating. $1 to $9 are filled correctly, above $9 they are
not filled correctly.

$10 had the contents of $1 plus a "0" added to the string,
same for $11 to $15 (the ones I tested with). So $14 has a "4" added.

Looking at the set docs it talks about $1 to $n, so I think values
above 9 should work.

Cheers Brian




 

 

[Marcel Nijenhof]

On Wed, Jul 16, 2008 at 10:06:36AM -0000, bloedmann999 wrote:
>
> Ah, seen that somewhere before too. I have one problem with that that
> I am investigating. $1 to $9 are filled correctly, above $9 they are
> not filled correctly.
>

You can solve that by writing curly brackets arround the variable
name (or in this case the number).

#!/bin/bash

A="a b c d e f g h i j k l m n o p q r s t u v w x y z"

set -- ${A}

echo "   1:     ${1}"
echo "   2:     ${2}"
echo "   3:     ${3}"
echo "   4:     ${4}"
echo "   5:     ${5}"
echo "   6:     ${6}"
echo "   7:     ${7}"
echo "   8:     ${8}"
echo "   9:     ${9}"
echo "  10:     ${10}"
echo "  11:     ${11}"
echo "  12:     ${12}"
echo "  13:     ${13}"
echo "  14:     ${14}"
echo "  15:     ${15}"
echo "  16:     ${16}"
echo "  17:     ${17}"
echo "  18:     ${18}"
echo "  19:     ${19}"
echo "  20:     ${20}"
echo "  21:     ${21}"
echo "  22:     ${22}"
echo "  23:     ${23}"
echo "  24:     ${24}"
echo "  25:     ${25}"
echo "  26:     ${26}"


--
Marcel Nijenhof
Optware package developer
nslu2@...

[Brian-253]

--- In nslu2-linux@..., nslu2@... wrote:

>
> On Wed, Jul 16, 2008 at 10:06:36AM -0000, bloedmann999 wrote:
> >
> > Ah, seen that somewhere before too. I have one problem with that that
> > I am investigating. $1 to $9 are filled correctly, above $9 they are
> > not filled correctly.
> >
>
> You can solve that by writing curly brackets arround the variable
> name (or in this case the number).
>
> #!/bin/bash
>
> A="a b c d e f g h i j k l m n o p q r s t u v w x y z"
>
> set -- ${A}
>
> echo "   1:     ${1}"
> echo "   2:     ${2}"
> echo "   3:     ${3}"
> echo "   4:     ${4}"
> echo "   5:     ${5}"
> echo "   6:     ${6}"
> echo "   7:     ${7}"
> echo "   8:     ${8}"
> echo "   9:     ${9}"
> echo "  10:     ${10}"
> echo "  11:     ${11}"
> echo "  12:     ${12}"
> echo "  13:     ${13}"
> echo "  14:     ${14}"
> echo "  15:     ${15}"
> echo "  16:     ${16}"
> echo "  17:     ${17}"
> echo "  18:     ${18}"
> echo "  19:     ${19}"
> echo "  20:     ${20}"
> echo "  21:     ${21}"
> echo "  22:     ${22}"
> echo "  23:     ${23}"
> echo "  24:     ${24}"
> echo "  25:     ${25}"
> echo "  26:     ${26}"
>
>
> --
> Marcel Nijenhof
> Optware package developer
> nslu2@...
>

Marcel,
thanks that works OK. Looking at the book I have on Bash I can only
see the the following that fits to this:

B=${A:start:length}

is that what this is doing, but as the start and length aren't there,
then the complete string is taken? Just wondering why bash works
differently for variables up to 9 and from 10 onwards.

Cheers Brian

[Marcel Nijenhof]

On Sat, 2008-07-19 at 10:22 +0000, bloedmann999 wrote:

> thanks that works OK. Looking at the book I have on Bash I can only
> see the the following that fits to this:
>
> B=${A:start:length}

You are allowed to write any variable with curly brackets.
It's just an other way of writing the variables.

But there are a lot of extra features if you use the curly
brackets combined with some extra operators.

e.g.:
        ${VAR/search/replace}

But there are a lot more possibilities.

--
Marcel Nijenhof
Optware package developer
nslu2@...

[jl.050877]

On Sat, Jul 19, 2008 at 10:22:00AM -0000, bloedmann999 wrote:
> Marcel,
> thanks that works OK. Looking at the book I have on Bash I can only
> see the the following that fits to this:
>
> B=${A:start:length}
>
> is that what this is doing, but as the start and length aren't there,
> then the complete string is taken? Just wondering why bash works
> differently for variables up to 9 and from 10 onwards.

Look at the man page for Bash (on the web or on any system that has
man installed) and it explains this, including the problem with
two-digit parameters names.  From the section entitled "Parameter
Expansion":

    ${parameter}
       The  value  of  parameter  is  substituted.   The braces  are
       required when parameter is a positional  parameter with  more
       than  one  digit, or when parameter is followed by a character
       which is not to be interpreted as part of its name.

John

[Brian-253]

--- In nslu2-linux@..., jl.050877@... wrote:

>
> On Sat, Jul 19, 2008 at 10:22:00AM -0000, bloedmann999 wrote:
> > Marcel,
> > thanks that works OK. Looking at the book I have on Bash I can only
> > see the the following that fits to this:
> >
> > B=${A:start:length}
> >
> > is that what this is doing, but as the start and length aren't there,
> > then the complete string is taken? Just wondering why bash works
> > differently for variables up to 9 and from 10 onwards.
>
> Look at the man page for Bash (on the web or on any system that has
> man installed) and it explains this, including the problem with
> two-digit parameters names.  From the section entitled "Parameter
> Expansion":
>
>     ${parameter}
>        The  value  of  parameter  is  substituted.   The braces  are
>        required when parameter is a positional  parameter with  more
>        than  one  digit, or when parameter is followed by a character
>        which is not to be interpreted as part of its name.
>
> John
>

Thanks I have that all working correctly now on debian and unslung. I
would like to hijack my own thead for a similar question on Bash.
I occasionally get a string passed to my script that has things like
this in it; `UNKNOWN'.
And that is causing me problems. So I though I would just translate
the characters I did not need to blanks, but the string above always
causes a syntax error of some kind or another. Looking at the Bash manual:
3.1.2.2 Single Quotes

Enclosing characters in single quotes (`'') preserves the literal
value of each character within the quotes. A single quote may not
occur between single quotes, even when preceded by a backslash.

So a single quote may not appear in the string, in mine it does. Also
I have no chance to translate to add backslashes, as it breaks
straight away.

Single quotes:
# ./parse-msgs-2.sh test '`UNKNOWN''
Syntax error: Unterminated quoted string

Double quotes:
# ./parse-msgs-2.sh test "`UNKNOWN'"
Syntax error: EOF in backquote substitution

From anothe point of view, the message itself is being produced by
login, for example:

Jul 25 12:40:55 SLUG1 login[3865]: invalid password for `UNKNOWN' on
`ttyp3' from `192.168.123.166'

Maybe there is a way to stop the message being produced with these
single quotes and ticks?

OTOH I would be very interested in a method to allow me to process
such a string if possible.

The testing of this was done on a Firmware Version:
V2.3R63-uNSLUng-6.8-beta, with bash - 3.2.33-1.

Cheers Brian






 

[Mike (mwester)]

bloedmann999 wrote:

> --- In nslu2-linux@..., jl.050877@... wrote:
>> On Sat, Jul 19, 2008 at 10:22:00AM -0000, bloedmann999 wrote:
>>> Marcel,
>>> thanks that works OK. Looking at the book I have on Bash I can only
>>> see the the following that fits to this:
>>>
>>> B=${A:start:length}
>>>
>>> is that what this is doing, but as the start and length aren't there,
>>> then the complete string is taken? Just wondering why bash works
>>> differently for variables up to 9 and from 10 onwards.
>> Look at the man page for Bash (on the web or on any system that has
>> man installed) and it explains this, including the problem with
>> two-digit parameters names.  From the section entitled "Parameter
>> Expansion":
>>
>>     ${parameter}
>>        The  value  of  parameter  is  substituted.   The braces  are
>>        required when parameter is a positional  parameter with  more
>>        than  one  digit, or when parameter is followed by a character
>>        which is not to be interpreted as part of its name.
>>
>> John
>>
> Thanks I have that all working correctly now on debian and unslung. I
> would like to hijack my own thead for a similar question on Bash.
> I occasionally get a string passed to my script that has things like
> this in it; `UNKNOWN'.
> And that is causing me problems. So I though I would just translate
> the characters I did not need to blanks, but the string above always
> causes a syntax error of some kind or another. Looking at the Bash manual:
> 3.1.2.2 Single Quotes
>
> Enclosing characters in single quotes (`'') preserves the literal
> value of each character within the quotes. A single quote may not
> occur between single quotes, even when preceded by a backslash.
>
> So a single quote may not appear in the string, in mine it does. Also
> I have no chance to translate to add backslashes, as it breaks
> straight away.
>
> Single quotes:
> # ./parse-msgs-2.sh test '`UNKNOWN''
> Syntax error: Unterminated quoted string
>
> Double quotes:
> # ./parse-msgs-2.sh test "`UNKNOWN'"
> Syntax error: EOF in backquote substitution
>
>>From anothe point of view, the message itself is being produced by
> login, for example:
>
> Jul 25 12:40:55 SLUG1 login[3865]: invalid password for `UNKNOWN' on
> `ttyp3' from `192.168.123.166'
>
> Maybe there is a way to stop the message being produced with these
> single quotes and ticks?
>
> OTOH I would be very interested in a method to allow me to process
> such a string if possible.
>
> The testing of this was done on a Firmware Version:
> V2.3R63-uNSLUng-6.8-beta, with bash - 3.2.33-1.
>
> Cheers Brian

When you begin to encounter issues like this, it's usually an indication
that your application has exceeded the ability of the language -- I'd
start looking into using awk to parse the messages so that by the time
bash gets the string, it's all cleaned up.  Perhaps I'm showing my age,
but I've always preferred to use awk and sed to do the string
manipulation rather than the funky bash variable operations.

Note that the built-in awk is the busybox awk, which is limited (and
slightly buggy) compared to the full GNU awk (gawk).  I've had less
trouble with the busybox version of sed, but that might be because I
gravitate towards awk rather than use advanced sed features.

Mike (mwester)

[Brian-253]

--- In nslu2-linux@..., "Mike (mwester)" <mwester@...> wrote:

>
> bloedmann999 wrote:
> > --- In nslu2-linux@..., jl.050877@ wrote:
> >> On Sat, Jul 19, 2008 at 10:22:00AM -0000, bloedmann999 wrote:
> >>> Marcel,
> >>> thanks that works OK. Looking at the book I have on Bash I can only
> >>> see the the following that fits to this:
> >>>
> >>> B=${A:start:length}
> >>>
> >>> is that what this is doing, but as the start and length aren't

there,

> >>> then the complete string is taken? Just wondering why bash works
> >>> differently for variables up to 9 and from 10 onwards.
> >> Look at the man page for Bash (on the web or on any system that has
> >> man installed) and it explains this, including the problem with
> >> two-digit parameters names.  From the section entitled "Parameter
> >> Expansion":
> >>
> >>     ${parameter}
> >>        The  value  of  parameter  is  substituted.   The braces  are
> >>        required when parameter is a positional  parameter with  more
> >>        than  one  digit, or when parameter is followed by a character
> >>        which is not to be interpreted as part of its name.
> >>
> >> John
> >>
> > Thanks I have that all working correctly now on debian and unslung. I
> > would like to hijack my own thead for a similar question on Bash.
> > I occasionally get a string passed to my script that has things like
> > this in it; `UNKNOWN'.
> > And that is causing me problems. So I though I would just translate
> > the characters I did not need to blanks, but the string above always
> > causes a syntax error of some kind or another. Looking at the Bash

manual:

> > 3.1.2.2 Single Quotes
> >
> > Enclosing characters in single quotes (`'') preserves the literal
> > value of each character within the quotes. A single quote may not
> > occur between single quotes, even when preceded by a backslash.
> >
> > So a single quote may not appear in the string, in mine it does. Also
> > I have no chance to translate to add backslashes, as it breaks
> > straight away.
> >
> > Single quotes:
> > # ./parse-msgs-2.sh test '`UNKNOWN''
> > Syntax error: Unterminated quoted string
> >
> > Double quotes:
> > # ./parse-msgs-2.sh test "`UNKNOWN'"
> > Syntax error: EOF in backquote substitution
> >
> >>From anothe point of view, the message itself is being produced by
> > login, for example:
> >
> > Jul 25 12:40:55 SLUG1 login[3865]: invalid password for `UNKNOWN' on
> > `ttyp3' from `192.168.123.166'
> >
> > Maybe there is a way to stop the message being produced with these
> > single quotes and ticks?
> >
> > OTOH I would be very interested in a method to allow me to process
> > such a string if possible.
> >
> > The testing of this was done on a Firmware Version:
> > V2.3R63-uNSLUng-6.8-beta, with bash - 3.2.33-1.
> >
> > Cheers Brian
>
> When you begin to encounter issues like this, it's usually an indication
> that your application has exceeded the ability of the language -- I'd
> start looking into using awk to parse the messages so that by the time
> bash gets the string, it's all cleaned up.  Perhaps I'm showing my age,
> but I've always preferred to use awk and sed to do the string
> manipulation rather than the funky bash variable operations.
>
> Note that the built-in awk is the busybox awk, which is limited (and
> slightly buggy) compared to the full GNU awk (gawk).  I've had less
> trouble with the busybox version of sed, but that might be because I
> gravitate towards awk rather than use advanced sed features.
>
> Mike (mwester)
>

Mike,

I understand what you are saying. I am calling a bash script with this
message, and was going to use awk or more likely sed to get rid of
those characters. I am not very proficient at scripting so was messing
around, in this case with "tr" to translate the special characters to
blanks, but found that I immediately get errors, as soon as such a
string is used in the bash script. To me they look like the bash
parser that is complaining.

So, I guess what I really need to know, is how can I encapsulate such
a message, so that I would be able to pass it to an external program
without bash complaining beforehand?

Thanks

Cheers Brian  

[jl.050877]

Brian,

     When bash quoting problems arise, I tend to switch to perl or
python.  Perl was originally designed just for your purpose:
analyzing log files.  For code that takes more than just a few
lines, I prefer python over perl because of, in my opinion, its
better readability and maintainability.  YMMV.

     In either perl or python, you open the log file for input (or
send it to the scripts stdin) and all bash quoting issues disappear.

> > > Single quotes:
> > > # ./parse-msgs-2.sh test '`UNKNOWN''
> > > Syntax error: Unterminated quoted string
> > >
> > > Double quotes:
> > > # ./parse-msgs-2.sh test "`UNKNOWN'"
> > > Syntax error: EOF in backquote substitution

You didn't explain why you want to do it this way.  Could you instead
use something like:

tail -f /var/log/messages | mybashscript.sh

and use bash's read (inside a loop) to capture the messages for
scrubbing and processing.  Because the messages pass through stdin
and thus never appear on the command line, bash doesn't try to
match forward quotes or execute backquote pairs.

John

[Mike (mwester)]

bloedmann999 wrote:
...
> So, I guess what I really need to know, is how can I encapsulate such
> a message, so that I would be able to pass it to an external program
> without bash complaining beforehand?
>
> Thanks
>
> Cheers Brian  

I think in general you don't want to pass it about as an argument; the
problem is that the contents of an argument are not opaque to bash.
Instead, you might just place the message in a variable, and then try to
pass a reference to that variable about -- but that just defers the
inevitable.  Bash is a command interpretor; you need a language.

Perl?  Python?  These would work.

If you really must deal with this sort of thing, you need to encode the
original text in such a fashion that all special characters are taken
care of before they end up in a shell variable.  My first thought would
be to translate the message string to the appropriate HTML encoding --
it's standard.  That might be a bit much for awk, however.  But there's
no doubt that using awk to read and parse the original text before it
ends up in a variable in the first place is the way to do it.

Mike (mwester)

[Brian-253]

--- In nslu2-linux@..., jl.050877@... wrote:

>
> Brian,
>
>      When bash quoting problems arise, I tend to switch to perl or
> python.  Perl was originally designed just for your purpose:
> analyzing log files.  For code that takes more than just a few
> lines, I prefer python over perl because of, in my opinion, its
> better readability and maintainability.  YMMV.
>
>      In either perl or python, you open the log file for input (or
> send it to the scripts stdin) and all bash quoting issues disappear.
>
> > > > Single quotes:
> > > > # ./parse-msgs-2.sh test '`UNKNOWN''
> > > > Syntax error: Unterminated quoted string
> > > >
> > > > Double quotes:
> > > > # ./parse-msgs-2.sh test "`UNKNOWN'"
> > > > Syntax error: EOF in backquote substitution
>
> You didn't explain why you want to do it this way.  Could you instead
> use something like:
>
> tail -f /var/log/messages | mybashscript.sh
>
> and use bash's read (inside a loop) to capture the messages for
> scrubbing and processing.  Because the messages pass through stdin
> and thus never appear on the command line, bash doesn't try to
> match forward quotes or execute backquote pairs.
>
> John
>

John,

what I showed was just as a test, I use logdog to keep an eye on my
logs and it calls certain scripts depending upon what messages appear.
Thinking about it "tail -f " would probably work just as well.

I will certainly try what you suggest to see if bash does react
differently. But you (and Mike) gave me a different idea, why dont I
call awk straight away from logdog, have that scrub the message, and
then pass the result on to my bash script? After all, what I am doing
in bash is not supposed to very complex, so I probably could stay with
bash.

Today we are supposed to have thunderstorms, so it looks like the
ideal opportunity to spend some mor etime on this.

Thanks.

Brian

[Brian-253]

--- In nslu2-linux@..., "Mike (mwester)" <mwester@...> wrote:

>
> bloedmann999 wrote:
> ...
> > So, I guess what I really need to know, is how can I encapsulate such
> > a message, so that I would be able to pass it to an external program
> > without bash complaining beforehand?
> >
> > Thanks
> >
> > Cheers Brian  
>
> I think in general you don't want to pass it about as an argument; the
> problem is that the contents of an argument are not opaque to bash.
> Instead, you might just place the message in a variable, and then try to
> pass a reference to that variable about -- but that just defers the
> inevitable.  Bash is a command interpretor; you need a language.
>
> Perl?  Python?  These would work.
>
> If you really must deal with this sort of thing, you need to encode the
> original text in such a fashion that all special characters are taken
> care of before they end up in a shell variable.  My first thought would
> be to translate the message string to the appropriate HTML encoding --
> it's standard.  That might be a bit much for awk, however.  But there's
> no doubt that using awk to read and parse the original text before it
> ends up in a variable in the first place is the way to do it.
>
> Mike (mwester)
>

Mike,
OK (please see my previous post). I guess I needed a whole new take on
what I am doing. So instead of solving this problem I just need to try
 to avoid it occuring.

Thanks.

Brian

[Brian-253]

--- In nslu2-linux@..., "bloedmann999" <Brian_Dorling@...>
wrote:
>
> --- In nslu2-linux@..., "Mike (mwester)" <mwester@> wrote:
> >
> > bloedmann999 wrote:
> > ...
> > > So, I guess what I really need to know, is how can I encapsulate
such

> > > a message, so that I would be able to pass it to an external program
> > > without bash complaining beforehand?
> > >
> > > Thanks
> > >
> > > Cheers Brian  
> >
> > I think in general you don't want to pass it about as an argument; the
> > problem is that the contents of an argument are not opaque to bash.
> > Instead, you might just place the message in a variable, and then

try to
> > pass a reference to that variable about -- but that just defers the
> > inevitable.  Bash is a command interpretor; you need a language.
> >
> > Perl?  Python?  These would work.
> >
> > If you really must deal with this sort of thing, you need to
encode the
> > original text in such a fashion that all special characters are taken
> > care of before they end up in a shell variable.  My first thought
would
> > be to translate the message string to the appropriate HTML encoding --
> > it's standard.  That might be a bit much for awk, however.  But
there's

> > no doubt that using awk to read and parse the original text before it
> > ends up in a variable in the first place is the way to do it.
> >
> > Mike (mwester)
> >
> Mike,
> OK (please see my previous post). I guess I needed a whole new take on
> what I am doing. So instead of solving this problem I just need to try
>  to avoid it occuring.
>
> Thanks.
>
> Brian
>

Well, the solution was far easier that I had thought, once I knew that
I had to avoid hitting this problem in bash. I am using logdog.pl,
that calls my bash script, so I just added a translate into the perl
script at the point just before it calls the external script, so the
"`" and "'" are now gone totally.

Thanks for all the help.

Cheers Brian