Charsets, Gtkmm and MySQL

[Bernd Robertz-2]

Hello list,

I'm not really sure, but I hope My question is in the right place here.

I wrote a simple database app like this:

Connect to a MySQL server via libmysql++,
and then list the rows in a treeview.
Doubleclick on a row shows a window with some entries and a textview
with the content of the selected row.
The textview is associated with a type text in the DB.

Everything is fine.

So but I ran really into problems with the charsets.
german umlaute (ö, ä, etc.) and some other characters are glyphs in the
treeview. Ok, thats not the main problem.

That is: The entries in the detailed view still shows the glyphs for
some chars like ö, ä.
But on some rows, the represented textbuffer for textview is not
visible. Wich means the textview is in some rows empty.

I run My app from a console and I got these error messages:

:::::::::
Gtk-CRITICAL **: gtk_text_buffer_emit_insert: assertion `g_utf8_validate
(text, len, NULL)' failed
:::::::::

In other rows, the textbuffer is correct shown, exepct some glyphs.

If I print the textbuffer as a simple .c_str() in a console it will be
shown correctly (inklusive ä, ö, ü...).
(Tried to show the textview as a .c_str(), without any effect.)

The SQL connection is serverside-configured as UTF-8 Unicode.
The SQL table collation is UTF-8 General CI (Case Insensitive).
The Application runs on a DE_de@Euro (ISO-8859-15).

All characters are shown correctly in a PHP app wich does the same, just
only online. :-)

I hope someone could turn on a light above my head, why I don't get the
right charactes.

Thanks alot in advance

Bernd


_______________________________________________
gtkmm-list mailing list
gtkmm-list@...
http://mail.gnome.org/mailman/listinfo/gtkmm-list

[milosz derezynski]

Are you using Glib::ustring anywhere in the process of getting the data from the SQL database to the view?

Glib::ustring implicitly (and in my opinion still at least not well enough documented/hinted at) converts from UTF-8
to the current locale with operators << and >>. If you are using a stringstream anywhere in the process, or are
using operator<< or operator>> in any other way, this could then be the cause of the problem.

Otherwise, and in the above described case too actually it would help very much if you could post (if possible, all of)
the relevant code online.

Regards,
M.

On Sun, Dec 28, 2008 at 8:53 PM, Bernd Robertz <bernd.robertz@...> wrote:

Hello list,

I'm not really sure, but I hope My question is in the right place here.

I wrote a simple database app like this:

Connect to a MySQL server via libmysql++,
and then list the rows in a treeview.
Doubleclick on a row shows a window with some entries and a textview
with the content of the selected row.
The textview is associated with a type text in the DB.

Everything is fine.

So but I ran really into problems with the charsets.
german umlaute (ö, ä, etc.) and some other characters are glyphs in the
treeview. Ok, thats not the main problem.

That is: The entries in the detailed view still shows the glyphs for
some chars like ö, ä.
But on some rows, the represented textbuffer for textview is not
visible. Wich means the textview is in some rows empty.

I run My app from a console and I got these error messages:

:::::::::
Gtk-CRITICAL **: gtk_text_buffer_emit_insert: assertion `g_utf8_validate
(text, len, NULL)' failed
:::::::::

In other rows, the textbuffer is correct shown, exepct some glyphs.

If I print the textbuffer as a simple .c_str() in a console it will be
shown correctly (inklusive ä, ö, ü...).
(Tried to show the textview as a .c_str(), without any effect.)

The SQL connection is serverside-configured as UTF-8 Unicode.
The SQL table collation is UTF-8 General CI (Case Insensitive).
The Application runs on a DE_de@Euro (ISO-8859-15).

All characters are shown correctly in a PHP app wich does the same, just
only online. :-)

I hope someone could turn on a light above my head, why I don't get the
right charactes.

Thanks alot in advance

Bernd


_______________________________________________
gtkmm-list mailing list
gtkmm-list@...
http://mail.gnome.org/mailman/listinfo/gtkmm-list

--
Please note that according to the German law on data retention,
information on every electronic information exchange with me is
retained for a period of six months.
[Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung zufolge
jeder elektronische Kontakt mit mir sechs Monate lang gespeichert wird.]

_______________________________________________
gtkmm-list mailing list
gtkmm-list@...
http://mail.gnome.org/mailman/listinfo/gtkmm-list

[Bernd Robertz-2]

Thanks alot for your fast reply,

I wrote a simple MySQL-Class wich is derived from the libsql++ class.

In this class I used std::string to get the columns of a row.
The columns (strings) are stored in a vector<string> wich I return to
Glib::ustring in My treeview-data-window.

Hiere are some snippets:

:::::::::::::::::::::::: SNIP ::::::::::::::::::::::
database.cpp
(My simple MySQL database class.)

DataBase::Database()
{
        // db-connect
        // query-setup
        // other connect related stuff

        mysqlpp::StoreQueryResult result; // Setup the result object
        int rows; // Keep track of result counts
}

void DataBase::returnrow(int dataid)
{
        // Setup db-query with dataid to get requested db-row
        //
        mysqlpp::Query query = dbcon.query(queries.alldata); // DB Query
        result = query.store(); // Store the results
        rows = result.num_rows(); // Set the number of results
}

std::vector<std::string> DataBase::fetchdata(int row)
{
        std::vector<std::string> tmp; // Temp vector for return
       
        for (int i = 0; i < result.field_names()->size(); ++i)
        {
                // Every column will be stored in the temp vector
                tmp.push_back((std::string)result[row][i]);
        }
        return tmp; // Return the temp-vector with the [columns]=data
}

------------

datawindow.cpp
(The window in where the data is displayed after dbl-click on treeview)

DataWindow::DataWindow()
{
        // Widgets setup
        // Database connect via DataBase-Object
}

void DataWindow::getid(int dataid)
{
        // From the treeview I call this method to set the
        // requested row-id
        mysql.returnrow(dataid);

        // Call this method to setup the entries and textview
        setdata();
}

void DataWindow::setdata()
{
        l_dataid.set_text(mysql.fetchdata(0)[0]);
        Gtk::TreeModel::Row selrow = *(ls_cbox->prepend());
        selrow[c_mod.c_cat] = mysql.fetchdata(0)[1];
        cbox.set_active(0);

        // Here comes the problem, I get every column from My
        // database-object wich I store in the entries.
        e_name.set_text(mysql.fetchdata(0)[2]);
        e_number.set_text(mysql.fetchdata(0)[3]);
        e_user.set_text(mysql.fetchdata(0)[4]);
        e_pass.set_text(mysql.fetchdata(0)[5]);
        e_mail.set_text(mysql.fetchdata(0)[6]);
        e_url.set_text(mysql.fetchdata(0)[7]);
        e_other.set_text(mysql.fetchdata(0)[8]);

        // Here we have the main problem: the textview-buffer
        // Some rows are shown here, some not.
        b_descr->set_text(mysql.fetchdata(0)[9]);
}

::::::::::::::::::::::::: SNIP END ::::::::::::::::::::::::::::

That is the main code where the problem occours.

The DataWindow is called on doubleclick on a treeview-row.
With the call of the DataWindow I call the DataView::getid()
wich receive the database row-id.
Then I fill with the DataView::setdata() method the widgets
in the new window.

Some chars are, as I told, glyphs, but the textbuffer is the main
problem.

As far as I know, the entries and textbuffers are expecting
Glib::ustring, but I thought the conversion from std::string to
Glib::ustring is quite compatible between these types.
I used std::string in the database class, because I wanted to isolated
it from Gtkmm for other not Gtkmm-related projects.

Thank you for helping.

Bernd


Am Sonntag, den 28.12.2008, 22:26 +0100 schrieb Milosz Derezynski:

> Are you using Glib::ustring anywhere in the process of getting the
> data from the SQL database to the view?
>
> Glib::ustring implicitly (and in my opinion still at least not well
> enough documented/hinted at) converts from UTF-8
> to the current locale with operators << and >>. If you are using a
> stringstream anywhere in the process, or are
> using operator<< or operator>> in any other way, this could then be
> the cause of the problem.
>
> Otherwise, and in the above described case too actually it would help
> very much if you could post (if possible, all of)
> the relevant code online.
>
> Regards,
> M.
>
> On Sun, Dec 28, 2008 at 8:53 PM, Bernd Robertz
> <bernd.robertz@...> wrote:
>         Hello list,
>        
>         I'm not really sure, but I hope My question is in the right
>         place here.
>        
>         I wrote a simple database app like this:
>        
>         Connect to a MySQL server via libmysql++,
>         and then list the rows in a treeview.
>         Doubleclick on a row shows a window with some entries and a
>         textview
>         with the content of the selected row.
>         The textview is associated with a type text in the DB.
>        
>         Everything is fine.
>        
>         So but I ran really into problems with the charsets.
>         german umlaute (ö, ä, etc.) and some other characters are
>         glyphs in the
>         treeview. Ok, thats not the main problem.
>        
>         That is: The entries in the detailed view still shows the
>         glyphs for
>         some chars like ö, ä.
>         But on some rows, the represented textbuffer for textview is
>         not
>         visible. Wich means the textview is in some rows empty.
>        
>         I run My app from a console and I got these error messages:
>        
>         :::::::::
>         Gtk-CRITICAL **: gtk_text_buffer_emit_insert: assertion
>         `g_utf8_validate
>         (text, len, NULL)' failed
>         :::::::::
>        
>         In other rows, the textbuffer is correct shown, exepct some
>         glyphs.
>        
>         If I print the textbuffer as a simple .c_str() in a console it
>         will be
>         shown correctly (inklusive ä, ö, ü...).
>         (Tried to show the textview as a .c_str(), without any
>         effect.)
>        
>         The SQL connection is serverside-configured as UTF-8 Unicode.
>         The SQL table collation is UTF-8 General CI (Case
>         Insensitive).
>         The Application runs on a DE_de@Euro (ISO-8859-15).
>        
>         All characters are shown correctly in a PHP app wich does the
>         same, just
>         only online. :-)
>        
>         I hope someone could turn on a light above my head, why I
>         don't get the
>         right charactes.
>        
>         Thanks alot in advance
>        
>         Bernd
>        
>        
>         _______________________________________________
>         gtkmm-list mailing list
>         gtkmm-list@...
>         http://mail.gnome.org/mailman/listinfo/gtkmm-list
>
>
>
> --
> Please note that according to the German law on data retention,
> information on every electronic information exchange with me is
> retained for a period of six months.
> [Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung
> zufolge
> jeder elektronische Kontakt mit mir sechs Monate lang gespeichert
> wird.]

_______________________________________________
gtkmm-list mailing list
gtkmm-list@...
http://mail.gnome.org/mailman/listinfo/gtkmm-list

[milosz derezynski]

Of what type is 'result'?

Also, could you add, before this line:

   tmp.push_back((std::string)result[row][i]);

this code:

   Glib::ustring utf8_str(result[row][i]);
   g_message("UTF-8 valid: %d: %s", int(utf8_str.validate()), utf8_str.c_str());

and see what it gives?


M.

P.S.:
Yes std::string and ustring are generally compatible, with the main differences being that ustring works by-character not by-byte (e.g. find(), operator[], etc), ustring has methods for validation of UTF-8 and related functionality; the main potential culprit here is operator<< and operator>> of ustring. I had really big problems with these using boost::format() until i realised that internally, it's a stream to which the args are fed using operator<<() of ustring.

On Mon, Dec 29, 2008 at 12:01 AM, Bernd Robertz <bernd.robertz@...> wrote:

Thanks alot for your fast reply,

I wrote a simple MySQL-Class wich is derived from the libsql++ class.

In this class I used std::string to get the columns of a row.
The columns (strings) are stored in a vector<string> wich I return to
Glib::ustring in My treeview-data-window.

Hiere are some snippets:

:::::::::::::::::::::::: SNIP ::::::::::::::::::::::
database.cpp
(My simple MySQL database class.)

DataBase::Database()
{
       // db-connect
       // query-setup
       // other connect related stuff

       mysqlpp::StoreQueryResult result; // Setup the result object
       int rows; // Keep track of result counts
}

void DataBase::returnrow(int dataid)
{
       // Setup db-query with dataid to get requested db-row
       //
       mysqlpp::Query query = dbcon.query(queries.alldata); // DB Query
       result = query.store(); // Store the results
       rows = result.num_rows(); // Set the number of results
}

std::vector<std::string> DataBase::fetchdata(int row)
{
       std::vector<std::string> tmp; // Temp vector for return

       for (int i = 0; i < result.field_names()->size(); ++i)
       {
               // Every column will be stored in the temp vector
               tmp.push_back((std::string)result[row][i]);
       }
       return tmp; // Return the temp-vector with the [columns]=data
}

------------

datawindow.cpp
(The window in where the data is displayed after dbl-click on treeview)

DataWindow::DataWindow()
{
       // Widgets setup
       // Database connect via DataBase-Object
}

void DataWindow::getid(int dataid)
{
       // From the treeview I call this method to set the
       // requested row-id
       mysql.returnrow(dataid);

       // Call this method to setup the entries and textview
       setdata();
}

void DataWindow::setdata()
{
       l_dataid.set_text(mysql.fetchdata(0)[0]);
       Gtk::TreeModel::Row selrow = *(ls_cbox->prepend());
       selrow[c_mod.c_cat] = mysql.fetchdata(0)[1];
       cbox.set_active(0);

       // Here comes the problem, I get every column from My
       // database-object wich I store in the entries.
       e_name.set_text(mysql.fetchdata(0)[2]);
       e_number.set_text(mysql.fetchdata(0)[3]);
       e_user.set_text(mysql.fetchdata(0)[4]);
       e_pass.set_text(mysql.fetchdata(0)[5]);
       e_mail.set_text(mysql.fetchdata(0)[6]);
       e_url.set_text(mysql.fetchdata(0)[7]);
       e_other.set_text(mysql.fetchdata(0)[8]);

       // Here we have the main problem: the textview-buffer
       // Some rows are shown here, some not.
       b_descr->set_text(mysql.fetchdata(0)[9]);
}

::::::::::::::::::::::::: SNIP END ::::::::::::::::::::::::::::

That is the main code where the problem occours.

The DataWindow is called on doubleclick on a treeview-row.
With the call of the DataWindow I call the DataView::getid()
wich receive the database row-id.
Then I fill with the DataView::setdata() method the widgets
in the new window.

Some chars are, as I told, glyphs, but the textbuffer is the main
problem.

As far as I know, the entries and textbuffers are expecting
Glib::ustring, but I thought the conversion from std::string to
Glib::ustring is quite compatible between these types.
I used std::string in the database class, because I wanted to isolated
it from Gtkmm for other not Gtkmm-related projects.

Thank you for helping.

Bernd


Am Sonntag, den 28.12.2008, 22:26 +0100 schrieb Milosz Derezynski:

> Are you using Glib::ustring anywhere in the process of getting the
> data from the SQL database to the view?
>
> Glib::ustring implicitly (and in my opinion still at least not well
> enough documented/hinted at) converts from UTF-8
> to the current locale with operators << and >>. If you are using a
> stringstream anywhere in the process, or are
> using operator<< or operator>> in any other way, this could then be
> the cause of the problem.
>
> Otherwise, and in the above described case too actually it would help
> very much if you could post (if possible, all of)
> the relevant code online.
>
> Regards,
> M.
>
> On Sun, Dec 28, 2008 at 8:53 PM, Bernd Robertz
> <bernd.robertz@...> wrote:
>         Hello list,
>
>         I'm not really sure, but I hope My question is in the right
>         place here.
>
>         I wrote a simple database app like this:
>
>         Connect to a MySQL server via libmysql++,
>         and then list the rows in a treeview.
>         Doubleclick on a row shows a window with some entries and a
>         textview
>         with the content of the selected row.
>         The textview is associated with a type text in the DB.
>
>         Everything is fine.
>
>         So but I ran really into problems with the charsets.
>         german umlaute (ö, ä, etc.) and some other characters are
>         glyphs in the
>         treeview. Ok, thats not the main problem.
>
>         That is: The entries in the detailed view still shows the
>         glyphs for
>         some chars like ö, ä.
>         But on some rows, the represented textbuffer for textview is
>         not
>         visible. Wich means the textview is in some rows empty.
>
>         I run My app from a console and I got these error messages:
>
>         :::::::::
>         Gtk-CRITICAL **: gtk_text_buffer_emit_insert: assertion
>         `g_utf8_validate
>         (text, len, NULL)' failed
>         :::::::::
>
>         In other rows, the textbuffer is correct shown, exepct some
>         glyphs.
>
>         If I print the textbuffer as a simple .c_str() in a console it
>         will be
>         shown correctly (inklusive ä, ö, ü...).
>         (Tried to show the textview as a .c_str(), without any
>         effect.)
>
>         The SQL connection is serverside-configured as UTF-8 Unicode.
>         The SQL table collation is UTF-8 General CI (Case
>         Insensitive).
>         The Application runs on a DE_de@Euro (ISO-8859-15).
>
>         All characters are shown correctly in a PHP app wich does the
>         same, just
>         only online. :-)
>
>         I hope someone could turn on a light above my head, why I
>         don't get the
>         right charactes.
>
>         Thanks alot in advance
>
>         Bernd
>
>
>         _______________________________________________
>         gtkmm-list mailing list
>         gtkmm-list@...
>         http://mail.gnome.org/mailman/listinfo/gtkmm-list
>
>
>
> --
> Please note that according to the German law on data retention,
> information on every electronic information exchange with me is
> retained for a period of six months.
> [Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung
> zufolge
> jeder elektronische Kontakt mit mir sechs Monate lang gespeichert
> wird.]

--
Please note that according to the German law on data retention,
information on every electronic information exchange with me is
retained for a period of six months.
[Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung zufolge
jeder elektronische Kontakt mit mir sechs Monate lang gespeichert wird.]

_______________________________________________
gtkmm-list mailing list
gtkmm-list@...
http://mail.gnome.org/mailman/listinfo/gtkmm-list

[Bernd Robertz-2]

Hello,

thanks for the code.

result is an object of mysqlpp::StoreQueryResult in the sql-libs.
It stores the query data.
But in the docs of libmysql++ is described, that are the stored data
handeld as a derived std::string. Also compatible to the original
std::string.

I inserted your code as you described and it returns the following
output (2 textviews as example, one with shown data, one with empty)
The output is taken from console:

:::::::::::::::::: SNIP first example data :::::::::::::::::

** Message: UTF-8 valid: 0: Demozugang ist der gleiche.

Auch f\xfcr gesch\xe4ftliche Aktivit\xe4ten (Streeting
Unternehmensvertrieb, siehe Unterlagen).

Vertriebsnummer: 00028

Vertriebsseite:
URL: http://vertrieb.streeting.com
Benutzer: xxxx
Kennwort: xxxx

Abrechnungssystem (im Vertriebslogin):
Benutzer: xxxx
Kennwort: xxxx

:::::::::::::::::: SNIP second example data :::::::::::::::::

** Message: UTF-8 valid: 1: Daten CySoTec:
--------------

Jabba (XMPP) Messenger Client:
Benutzer: xxxx
Kennwort: xxxx
Server: cysotec.info
Jabba ID: xrobbi@...
[...]

:::::::::::::::::: SNIP example data end :::::::::::::::::

As far as I can see, the one with empty data is not UTF-8 valid
and the other is.
So, what can I do, to convert the Data from the SQL server to My app in
a valid UTF-8, or on the other hand,
what collation should I use in on My SQL server?

Thank you alot

Bernd


Am Montag, den 29.12.2008, 00:47 +0100 schrieb Milosz Derezynski:

> Of what type is 'result'?
>
> Also, could you add, before this line:
>
>    tmp.push_back((std::string)result[row][i]);
>
> this code:
>
>    Glib::ustring utf8_str(result[row][i]);
>    g_message("UTF-8 valid: %d: %s", int(utf8_str.validate()),
> utf8_str.c_str());
>
> and see what it gives?
>
>
> M.
>
> P.S.:
> Yes std::string and ustring are generally compatible, with the main
> differences being that ustring works by-character not by-byte (e.g.
> find(), operator[], etc), ustring has methods for validation of UTF-8
> and related functionality; the main potential culprit here is
> operator<< and operator>> of ustring. I had really big problems with
> these using boost::format() until i realised that internally, it's a
> stream to which the args are fed using operator<<() of ustring.
>
>
> On Mon, Dec 29, 2008 at 12:01 AM, Bernd Robertz
> <bernd.robertz@...> wrote:
>         Thanks alot for your fast reply,
>        
>         I wrote a simple MySQL-Class wich is derived from the libsql++
>         class.
>        
>         In this class I used std::string to get the columns of a row.
>         The columns (strings) are stored in a vector<string> wich I
>         return to
>         Glib::ustring in My treeview-data-window.
>        
>         Hiere are some snippets:
>        
>         :::::::::::::::::::::::: SNIP ::::::::::::::::::::::
>         database.cpp
>         (My simple MySQL database class.)
>        
>         DataBase::Database()
>         {
>                // db-connect
>                // query-setup
>                // other connect related stuff
>        
>                mysqlpp::StoreQueryResult result; // Setup the result
>         object
>                int rows; // Keep track of result counts
>         }
>        
>         void DataBase::returnrow(int dataid)
>         {
>                // Setup db-query with dataid to get requested db-row
>                //
>                mysqlpp::Query query = dbcon.query(queries.alldata); //
>         DB Query
>                result = query.store(); // Store the results
>                rows = result.num_rows(); // Set the number of results
>         }
>        
>         std::vector<std::string> DataBase::fetchdata(int row)
>         {
>                std::vector<std::string> tmp; // Temp vector for return
>        
>                for (int i = 0; i < result.field_names()->size(); ++i)
>                {
>                        // Every column will be stored in the temp
>         vector
>                        tmp.push_back((std::string)result[row][i]);
>                }
>                return tmp; // Return the temp-vector with the
>         [columns]=data
>         }
>        
>         ------------
>        
>         datawindow.cpp
>         (The window in where the data is displayed after dbl-click on
>         treeview)
>        
>         DataWindow::DataWindow()
>         {
>                // Widgets setup
>                // Database connect via DataBase-Object
>         }
>        
>         void DataWindow::getid(int dataid)
>         {
>                // From the treeview I call this method to set the
>                // requested row-id
>                mysql.returnrow(dataid);
>        
>                // Call this method to setup the entries and textview
>                setdata();
>         }
>        
>         void DataWindow::setdata()
>         {
>                l_dataid.set_text(mysql.fetchdata(0)[0]);
>                Gtk::TreeModel::Row selrow = *(ls_cbox->prepend());
>                selrow[c_mod.c_cat] = mysql.fetchdata(0)[1];
>                cbox.set_active(0);
>        
>                // Here comes the problem, I get every column from My
>                // database-object wich I store in the entries.
>                e_name.set_text(mysql.fetchdata(0)[2]);
>                e_number.set_text(mysql.fetchdata(0)[3]);
>                e_user.set_text(mysql.fetchdata(0)[4]);
>                e_pass.set_text(mysql.fetchdata(0)[5]);
>                e_mail.set_text(mysql.fetchdata(0)[6]);
>                e_url.set_text(mysql.fetchdata(0)[7]);
>                e_other.set_text(mysql.fetchdata(0)[8]);
>        
>                // Here we have the main problem: the textview-buffer
>                // Some rows are shown here, some not.
>                b_descr->set_text(mysql.fetchdata(0)[9]);
>         }
>        
>         ::::::::::::::::::::::::: SNIP
>         END ::::::::::::::::::::::::::::
>        
>         That is the main code where the problem occours.
>        
>         The DataWindow is called on doubleclick on a treeview-row.
>         With the call of the DataWindow I call the DataView::getid()
>         wich receive the database row-id.
>         Then I fill with the DataView::setdata() method the widgets
>         in the new window.
>        
>         Some chars are, as I told, glyphs, but the textbuffer is the
>         main
>         problem.
>        
>         As far as I know, the entries and textbuffers are expecting
>         Glib::ustring, but I thought the conversion from std::string
>         to
>         Glib::ustring is quite compatible between these types.
>         I used std::string in the database class, because I wanted to
>         isolated
>         it from Gtkmm for other not Gtkmm-related projects.
>        
>         Thank you for helping.
>        
>         Bernd
>        
>        
>         Am Sonntag, den 28.12.2008, 22:26 +0100 schrieb Milosz
>         Derezynski:
>        
>         > Are you using Glib::ustring anywhere in the process of
>         getting the
>         > data from the SQL database to the view?
>         >
>         > Glib::ustring implicitly (and in my opinion still at least
>         not well
>         > enough documented/hinted at) converts from UTF-8
>         > to the current locale with operators << and >>. If you are
>         using a
>         > stringstream anywhere in the process, or are
>         > using operator<< or operator>> in any other way, this could
>         then be
>         > the cause of the problem.
>         >
>         > Otherwise, and in the above described case too actually it
>         would help
>         > very much if you could post (if possible, all of)
>         > the relevant code online.
>         >
>         > Regards,
>         > M.
>         >
>         > On Sun, Dec 28, 2008 at 8:53 PM, Bernd Robertz
>         > <bernd.robertz@...> wrote:
>         >         Hello list,
>         >
>         >         I'm not really sure, but I hope My question is in
>         the right
>         >         place here.
>         >
>         >         I wrote a simple database app like this:
>         >
>         >         Connect to a MySQL server via libmysql++,
>         >         and then list the rows in a treeview.
>         >         Doubleclick on a row shows a window with some
>         entries and a
>         >         textview
>         >         with the content of the selected row.
>         >         The textview is associated with a type text in the
>         DB.
>         >
>         >         Everything is fine.
>         >
>         >         So but I ran really into problems with the charsets.
>         >         german umlaute (ö, ä, etc.) and some other
>         characters are
>         >         glyphs in the
>         >         treeview. Ok, thats not the main problem.
>         >
>         >         That is: The entries in the detailed view still
>         shows the
>         >         glyphs for
>         >         some chars like ö, ä.
>         >         But on some rows, the represented textbuffer for
>         textview is
>         >         not
>         >         visible. Wich means the textview is in some rows
>         empty.
>         >
>         >         I run My app from a console and I got these error
>         messages:
>         >
>         >         :::::::::
>         >         Gtk-CRITICAL **: gtk_text_buffer_emit_insert:
>         assertion
>         >         `g_utf8_validate
>         >         (text, len, NULL)' failed
>         >         :::::::::
>         >
>         >         In other rows, the textbuffer is correct shown,
>         exepct some
>         >         glyphs.
>         >
>         >         If I print the textbuffer as a simple .c_str() in a
>         console it
>         >         will be
>         >         shown correctly (inklusive ä, ö, ü...).
>         >         (Tried to show the textview as a .c_str(), without
>         any
>         >         effect.)
>         >
>         >         The SQL connection is serverside-configured as UTF-8
>         Unicode.
>         >         The SQL table collation is UTF-8 General CI (Case
>         >         Insensitive).
>         >         The Application runs on a DE_de@Euro (ISO-8859-15).
>         >
>         >         All characters are shown correctly in a PHP app wich
>         does the
>         >         same, just
>         >         only online. :-)
>         >
>         >         I hope someone could turn on a light above my head,
>         why I
>         >         don't get the
>         >         right charactes.
>         >
>         >         Thanks alot in advance
>         >
>         >         Bernd
>         >
>         >
>         >         _______________________________________________
>         >         gtkmm-list mailing list
>         >         gtkmm-list@...
>         >         http://mail.gnome.org/mailman/listinfo/gtkmm-list
>         >
>         >
>         >
>         > --
>         > Please note that according to the German law on data
>         retention,
>         > information on every electronic information exchange with me
>         is
>         > retained for a period of six months.
>         > [Bitte beachten Sie, dass dem Gesetz zur
>         Vorratsdatenspeicherung
>         > zufolge
>         > jeder elektronische Kontakt mit mir sechs Monate lang
>         gespeichert
>         > wird.]
>        
>        
>
>
>
> --
> Please note that according to the German law on data retention,
> information on every electronic information exchange with me is
> retained for a period of six months.
> [Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung
> zufolge
> jeder elektronische Kontakt mit mir sechs Monate lang gespeichert
> wird.]

_______________________________________________
gtkmm-list mailing list
gtkmm-list@...
http://mail.gnome.org/mailman/listinfo/gtkmm-list

[Bernd Robertz-2]

Hello all,

I hope everyone is well arrived in the new year.

I come back to my last post.
It seems to be, that I have some weired missconfiguration with my
charsets.
If I'm inserting a row from My gtkmm-app, it will be shown correctly in
My app, but not in other apps like my PHP webfrontend or phpMyAdmin.

I think I know how to fix it.

Has anyone suggestions wich charset should generally be used i. e. for
client and server apps (Gtkmm, webserver, db, etc.)?

Thanks everyone who give Me the flashlight to get the problem.

Many thanks

Bernd


Am Montag, den 29.12.2008, 13:09 +0100 schrieb Bernd Robertz:

> Hello,
>
> thanks for the code.
>
> result is an object of mysqlpp::StoreQueryResult in the sql-libs.
> It stores the query data.
> But in the docs of libmysql++ is described, that are the stored data
> handeld as a derived std::string. Also compatible to the original
> std::string.
>
> I inserted your code as you described and it returns the following
> output (2 textviews as example, one with shown data, one with empty)
> The output is taken from console:
>
> :::::::::::::::::: SNIP first example data :::::::::::::::::
>
> ** Message: UTF-8 valid: 0: Demozugang ist der gleiche.
>
> Auch f\xfcr gesch\xe4ftliche Aktivit\xe4ten (Streeting
> Unternehmensvertrieb, siehe Unterlagen).
>
> Vertriebsnummer: 00028
>
> Vertriebsseite:
> URL: http://vertrieb.streeting.com
> Benutzer: xxxx
> Kennwort: xxxx
>
> Abrechnungssystem (im Vertriebslogin):
> Benutzer: xxxx
> Kennwort: xxxx
>
> :::::::::::::::::: SNIP second example data :::::::::::::::::
>
> ** Message: UTF-8 valid: 1: Daten CySoTec:
> --------------
>
> Jabba (XMPP) Messenger Client:
> Benutzer: xxxx
> Kennwort: xxxx
> Server: cysotec.info
> Jabba ID: xrobbi@...
> [...]
>
> :::::::::::::::::: SNIP example data end :::::::::::::::::
>
> As far as I can see, the one with empty data is not UTF-8 valid
> and the other is.
> So, what can I do, to convert the Data from the SQL server to My app in
> a valid UTF-8, or on the other hand,
> what collation should I use in on My SQL server?
>
> Thank you alot
>
> Bernd
>
>
> Am Montag, den 29.12.2008, 00:47 +0100 schrieb Milosz Derezynski:
> > Of what type is 'result'?
> >
> > Also, could you add, before this line:
> >
> >    tmp.push_back((std::string)result[row][i]);
> >
> > this code:
> >
> >    Glib::ustring utf8_str(result[row][i]);
> >    g_message("UTF-8 valid: %d: %s", int(utf8_str.validate()),
> > utf8_str.c_str());
> >
> > and see what it gives?
> >
> >
> > M.
> >
> > P.S.:
> > Yes std::string and ustring are generally compatible, with the main
> > differences being that ustring works by-character not by-byte (e.g.
> > find(), operator[], etc), ustring has methods for validation of UTF-8
> > and related functionality; the main potential culprit here is
> > operator<< and operator>> of ustring. I had really big problems with
> > these using boost::format() until i realised that internally, it's a
> > stream to which the args are fed using operator<<() of ustring.
> >
> >
> > On Mon, Dec 29, 2008 at 12:01 AM, Bernd Robertz
> > <bernd.robertz@...> wrote:
> >         Thanks alot for your fast reply,
> >        
> >         I wrote a simple MySQL-Class wich is derived from the libsql++
> >         class.
> >        
> >         In this class I used std::string to get the columns of a row.
> >         The columns (strings) are stored in a vector<string> wich I
> >         return to
> >         Glib::ustring in My treeview-data-window.
> >        
> >         Hiere are some snippets:
> >        
> >         :::::::::::::::::::::::: SNIP ::::::::::::::::::::::
> >         database.cpp
> >         (My simple MySQL database class.)
> >        
> >         DataBase::Database()
> >         {
> >                // db-connect
> >                // query-setup
> >                // other connect related stuff
> >        
> >                mysqlpp::StoreQueryResult result; // Setup the result
> >         object
> >                int rows; // Keep track of result counts
> >         }
> >        
> >         void DataBase::returnrow(int dataid)
> >         {
> >                // Setup db-query with dataid to get requested db-row
> >                //
> >                mysqlpp::Query query = dbcon.query(queries.alldata); //
> >         DB Query
> >                result = query.store(); // Store the results
> >                rows = result.num_rows(); // Set the number of results
> >         }
> >        
> >         std::vector<std::string> DataBase::fetchdata(int row)
> >         {
> >                std::vector<std::string> tmp; // Temp vector for return
> >        
> >                for (int i = 0; i < result.field_names()->size(); ++i)
> >                {
> >                        // Every column will be stored in the temp
> >         vector
> >                        tmp.push_back((std::string)result[row][i]);
> >                }
> >                return tmp; // Return the temp-vector with the
> >         [columns]=data
> >         }
> >        
> >         ------------
> >        
> >         datawindow.cpp
> >         (The window in where the data is displayed after dbl-click on
> >         treeview)
> >        
> >         DataWindow::DataWindow()
> >         {
> >                // Widgets setup
> >                // Database connect via DataBase-Object
> >         }
> >        
> >         void DataWindow::getid(int dataid)
> >         {
> >                // From the treeview I call this method to set the
> >                // requested row-id
> >                mysql.returnrow(dataid);
> >        
> >                // Call this method to setup the entries and textview
> >                setdata();
> >         }
> >        
> >         void DataWindow::setdata()
> >         {
> >                l_dataid.set_text(mysql.fetchdata(0)[0]);
> >                Gtk::TreeModel::Row selrow = *(ls_cbox->prepend());
> >                selrow[c_mod.c_cat] = mysql.fetchdata(0)[1];
> >                cbox.set_active(0);
> >        
> >                // Here comes the problem, I get every column from My
> >                // database-object wich I store in the entries.
> >                e_name.set_text(mysql.fetchdata(0)[2]);
> >                e_number.set_text(mysql.fetchdata(0)[3]);
> >                e_user.set_text(mysql.fetchdata(0)[4]);
> >                e_pass.set_text(mysql.fetchdata(0)[5]);
> >                e_mail.set_text(mysql.fetchdata(0)[6]);
> >                e_url.set_text(mysql.fetchdata(0)[7]);
> >                e_other.set_text(mysql.fetchdata(0)[8]);
> >        
> >                // Here we have the main problem: the textview-buffer
> >                // Some rows are shown here, some not.
> >                b_descr->set_text(mysql.fetchdata(0)[9]);
> >         }
> >        
> >         ::::::::::::::::::::::::: SNIP
> >         END ::::::::::::::::::::::::::::
> >        
> >         That is the main code where the problem occours.
> >        
> >         The DataWindow is called on doubleclick on a treeview-row.
> >         With the call of the DataWindow I call the DataView::getid()
> >         wich receive the database row-id.
> >         Then I fill with the DataView::setdata() method the widgets
> >         in the new window.
> >        
> >         Some chars are, as I told, glyphs, but the textbuffer is the
> >         main
> >         problem.
> >        
> >         As far as I know, the entries and textbuffers are expecting
> >         Glib::ustring, but I thought the conversion from std::string
> >         to
> >         Glib::ustring is quite compatible between these types.
> >         I used std::string in the database class, because I wanted to
> >         isolated
> >         it from Gtkmm for other not Gtkmm-related projects.
> >        
> >         Thank you for helping.
> >        
> >         Bernd
> >        
> >        
> >         Am Sonntag, den 28.12.2008, 22:26 +0100 schrieb Milosz
> >         Derezynski:
> >        
> >         > Are you using Glib::ustring anywhere in the process of
> >         getting the
> >         > data from the SQL database to the view?
> >         >
> >         > Glib::ustring implicitly (and in my opinion still at least
> >         not well
> >         > enough documented/hinted at) converts from UTF-8
> >         > to the current locale with operators << and >>. If you are
> >         using a
> >         > stringstream anywhere in the process, or are
> >         > using operator<< or operator>> in any other way, this could
> >         then be
> >         > the cause of the problem.
> >         >
> >         > Otherwise, and in the above described case too actually it
> >         would help
> >         > very much if you could post (if possible, all of)
> >         > the relevant code online.
> >         >
> >         > Regards,
> >         > M.
> >         >
> >         > On Sun, Dec 28, 2008 at 8:53 PM, Bernd Robertz
> >         > <bernd.robertz@...> wrote:
> >         >         Hello list,
> >         >
> >         >         I'm not really sure, but I hope My question is in
> >         the right
> >         >         place here.
> >         >
> >         >         I wrote a simple database app like this:
> >         >
> >         >         Connect to a MySQL server via libmysql++,
> >         >         and then list the rows in a treeview.
> >         >         Doubleclick on a row shows a window with some
> >         entries and a
> >         >         textview
> >         >         with the content of the selected row.
> >         >         The textview is associated with a type text in the
> >         DB.
> >         >
> >         >         Everything is fine.
> >         >
> >         >         So but I ran really into problems with the charsets.
> >         >         german umlaute (ö, ä, etc.) and some other
> >         characters are
> >         >         glyphs in the
> >         >         treeview. Ok, thats not the main problem.
> >         >
> >         >         That is: The entries in the detailed view still
> >         shows the
> >         >         glyphs for
> >         >         some chars like ö, ä.
> >         >         But on some rows, the represented textbuffer for
> >         textview is
> >         >         not
> >         >         visible. Wich means the textview is in some rows
> >         empty.
> >         >
> >         >         I run My app from a console and I got these error
> >         messages:
> >         >
> >         >         :::::::::
> >         >         Gtk-CRITICAL **: gtk_text_buffer_emit_insert:
> >         assertion
> >         >         `g_utf8_validate
> >         >         (text, len, NULL)' failed
> >         >         :::::::::
> >         >
> >         >         In other rows, the textbuffer is correct shown,
> >         exepct some
> >         >         glyphs.
> >         >
> >         >         If I print the textbuffer as a simple .c_str() in a
> >         console it
> >         >         will be
> >         >         shown correctly (inklusive ä, ö, ü...).
> >         >         (Tried to show the textview as a .c_str(), without
> >         any
> >         >         effect.)
> >         >
> >         >         The SQL connection is serverside-configured as UTF-8
> >         Unicode.
> >         >         The SQL table collation is UTF-8 General CI (Case
> >         >         Insensitive).
> >         >         The Application runs on a DE_de@Euro (ISO-8859-15).
> >         >
> >         >         All characters are shown correctly in a PHP app wich
> >         does the
> >         >         same, just
> >         >         only online. :-)
> >         >
> >         >         I hope someone could turn on a light above my head,
> >         why I
> >         >         don't get the
> >         >         right charactes.
> >         >
> >         >         Thanks alot in advance
> >         >
> >         >         Bernd
> >         >
> >         >
> >         >         _______________________________________________
> >         >         gtkmm-list mailing list
> >         >         gtkmm-list@...
> >         >         http://mail.gnome.org/mailman/listinfo/gtkmm-list
> >         >
> >         >
> >         >
> >         > --
> >         > Please note that according to the German law on data
> >         retention,
> >         > information on every electronic information exchange with me
> >         is
> >         > retained for a period of six months.
> >         > [Bitte beachten Sie, dass dem Gesetz zur
> >         Vorratsdatenspeicherung
> >         > zufolge
> >         > jeder elektronische Kontakt mit mir sechs Monate lang
> >         gespeichert
> >         > wird.]
> >        
> >        
> >
> >
> >
> > --
> > Please note that according to the German law on data retention,
> > information on every electronic information exchange with me is
> > retained for a period of six months.
> > [Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung
> > zufolge
> > jeder elektronische Kontakt mit mir sechs Monate lang gespeichert
> > wird.]
>
> _______________________________________________
> gtkmm-list mailing list
> gtkmm-list@...
> http://mail.gnome.org/mailman/listinfo/gtkmm-list

_______________________________________________
gtkmm-list mailing list
gtkmm-list@...
http://mail.gnome.org/mailman/listinfo/gtkmm-list

[Bernd Robertz-2]

Hello again,

I fixed the problem.

I know why I don't receive an answer about that. The solution was so
easy that it was a bit embrassing. :-)

Sorry for bothering and thanks to everyone who helped Me out.

Greetings

Bernd


Am Freitag, den 02.01.2009, 14:30 +0100 schrieb Bernd Robertz:

> Hello all,
>
> I hope everyone is well arrived in the new year.
>
> I come back to my last post.
> It seems to be, that I have some weired missconfiguration with my
> charsets.
> If I'm inserting a row from My gtkmm-app, it will be shown correctly in
> My app, but not in other apps like my PHP webfrontend or phpMyAdmin.
>
> I think I know how to fix it.
>
> Has anyone suggestions wich charset should generally be used i. e. for
> client and server apps (Gtkmm, webserver, db, etc.)?
>
> Thanks everyone who give Me the flashlight to get the problem.
>
> Many thanks
>
> Bernd
>
>
> Am Montag, den 29.12.2008, 13:09 +0100 schrieb Bernd Robertz:
> > Hello,
> >
> > thanks for the code.
> >
> > result is an object of mysqlpp::StoreQueryResult in the sql-libs.
> > It stores the query data.
> > But in the docs of libmysql++ is described, that are the stored data
> > handeld as a derived std::string. Also compatible to the original
> > std::string.
> >
> > I inserted your code as you described and it returns the following
> > output (2 textviews as example, one with shown data, one with empty)
> > The output is taken from console:
> >
> > :::::::::::::::::: SNIP first example data :::::::::::::::::
> >
> > ** Message: UTF-8 valid: 0: Demozugang ist der gleiche.
> >
> > Auch f\xfcr gesch\xe4ftliche Aktivit\xe4ten (Streeting
> > Unternehmensvertrieb, siehe Unterlagen).
> >
> > Vertriebsnummer: 00028
> >
> > Vertriebsseite:
> > URL: http://vertrieb.streeting.com
> > Benutzer: xxxx
> > Kennwort: xxxx
> >
> > Abrechnungssystem (im Vertriebslogin):
> > Benutzer: xxxx
> > Kennwort: xxxx
> >
> > :::::::::::::::::: SNIP second example data :::::::::::::::::
> >
> > ** Message: UTF-8 valid: 1: Daten CySoTec:
> > --------------
> >
> > Jabba (XMPP) Messenger Client:
> > Benutzer: xxxx
> > Kennwort: xxxx
> > Server: cysotec.info
> > Jabba ID: xrobbi@...
> > [...]
> >
> > :::::::::::::::::: SNIP example data end :::::::::::::::::
> >
> > As far as I can see, the one with empty data is not UTF-8 valid
> > and the other is.
> > So, what can I do, to convert the Data from the SQL server to My app in
> > a valid UTF-8, or on the other hand,
> > what collation should I use in on My SQL server?
> >
> > Thank you alot
> >
> > Bernd
> >
> >
> > Am Montag, den 29.12.2008, 00:47 +0100 schrieb Milosz Derezynski:
> > > Of what type is 'result'?
> > >
> > > Also, could you add, before this line:
> > >
> > >    tmp.push_back((std::string)result[row][i]);
> > >
> > > this code:
> > >
> > >    Glib::ustring utf8_str(result[row][i]);
> > >    g_message("UTF-8 valid: %d: %s", int(utf8_str.validate()),
> > > utf8_str.c_str());
> > >
> > > and see what it gives?
> > >
> > >
> > > M.
> > >
> > > P.S.:
> > > Yes std::string and ustring are generally compatible, with the main
> > > differences being that ustring works by-character not by-byte (e.g.
> > > find(), operator[], etc), ustring has methods for validation of UTF-8
> > > and related functionality; the main potential culprit here is
> > > operator<< and operator>> of ustring. I had really big problems with
> > > these using boost::format() until i realised that internally, it's a
> > > stream to which the args are fed using operator<<() of ustring.
> > >
> > >
> > > On Mon, Dec 29, 2008 at 12:01 AM, Bernd Robertz
> > > <bernd.robertz@...> wrote:
> > >         Thanks alot for your fast reply,
> > >        
> > >         I wrote a simple MySQL-Class wich is derived from the libsql++
> > >         class.
> > >        
> > >         In this class I used std::string to get the columns of a row.
> > >         The columns (strings) are stored in a vector<string> wich I
> > >         return to
> > >         Glib::ustring in My treeview-data-window.
> > >        
> > >         Hiere are some snippets:
> > >        
> > >         :::::::::::::::::::::::: SNIP ::::::::::::::::::::::
> > >         database.cpp
> > >         (My simple MySQL database class.)
> > >        
> > >         DataBase::Database()
> > >         {
> > >                // db-connect
> > >                // query-setup
> > >                // other connect related stuff
> > >        
> > >                mysqlpp::StoreQueryResult result; // Setup the result
> > >         object
> > >                int rows; // Keep track of result counts
> > >         }
> > >        
> > >         void DataBase::returnrow(int dataid)
> > >         {
> > >                // Setup db-query with dataid to get requested db-row
> > >                //
> > >                mysqlpp::Query query = dbcon.query(queries.alldata); //
> > >         DB Query
> > >                result = query.store(); // Store the results
> > >                rows = result.num_rows(); // Set the number of results
> > >         }
> > >        
> > >         std::vector<std::string> DataBase::fetchdata(int row)
> > >         {
> > >                std::vector<std::string> tmp; // Temp vector for return
> > >        
> > >                for (int i = 0; i < result.field_names()->size(); ++i)
> > >                {
> > >                        // Every column will be stored in the temp
> > >         vector
> > >                        tmp.push_back((std::string)result[row][i]);
> > >                }
> > >                return tmp; // Return the temp-vector with the
> > >         [columns]=data
> > >         }
> > >        
> > >         ------------
> > >        
> > >         datawindow.cpp
> > >         (The window in where the data is displayed after dbl-click on
> > >         treeview)
> > >        
> > >         DataWindow::DataWindow()
> > >         {
> > >                // Widgets setup
> > >                // Database connect via DataBase-Object
> > >         }
> > >        
> > >         void DataWindow::getid(int dataid)
> > >         {
> > >                // From the treeview I call this method to set the
> > >                // requested row-id
> > >                mysql.returnrow(dataid);
> > >        
> > >                // Call this method to setup the entries and textview
> > >                setdata();
> > >         }
> > >        
> > >         void DataWindow::setdata()
> > >         {
> > >                l_dataid.set_text(mysql.fetchdata(0)[0]);
> > >                Gtk::TreeModel::Row selrow = *(ls_cbox->prepend());
> > >                selrow[c_mod.c_cat] = mysql.fetchdata(0)[1];
> > >                cbox.set_active(0);
> > >        
> > >                // Here comes the problem, I get every column from My
> > >                // database-object wich I store in the entries.
> > >                e_name.set_text(mysql.fetchdata(0)[2]);
> > >                e_number.set_text(mysql.fetchdata(0)[3]);
> > >                e_user.set_text(mysql.fetchdata(0)[4]);
> > >                e_pass.set_text(mysql.fetchdata(0)[5]);
> > >                e_mail.set_text(mysql.fetchdata(0)[6]);
> > >                e_url.set_text(mysql.fetchdata(0)[7]);
> > >                e_other.set_text(mysql.fetchdata(0)[8]);
> > >        
> > >                // Here we have the main problem: the textview-buffer
> > >                // Some rows are shown here, some not.
> > >                b_descr->set_text(mysql.fetchdata(0)[9]);
> > >         }
> > >        
> > >         ::::::::::::::::::::::::: SNIP
> > >         END ::::::::::::::::::::::::::::
> > >        
> > >         That is the main code where the problem occours.
> > >        
> > >         The DataWindow is called on doubleclick on a treeview-row.
> > >         With the call of the DataWindow I call the DataView::getid()
> > >         wich receive the database row-id.
> > >         Then I fill with the DataView::setdata() method the widgets
> > >         in the new window.
> > >        
> > >         Some chars are, as I told, glyphs, but the textbuffer is the
> > >         main
> > >         problem.
> > >        
> > >         As far as I know, the entries and textbuffers are expecting
> > >         Glib::ustring, but I thought the conversion from std::string
> > >         to
> > >         Glib::ustring is quite compatible between these types.
> > >         I used std::string in the database class, because I wanted to
> > >         isolated
> > >         it from Gtkmm for other not Gtkmm-related projects.
> > >        
> > >         Thank you for helping.
> > >        
> > >         Bernd
> > >        
> > >        
> > >         Am Sonntag, den 28.12.2008, 22:26 +0100 schrieb Milosz
> > >         Derezynski:
> > >        
> > >         > Are you using Glib::ustring anywhere in the process of
> > >         getting the
> > >         > data from the SQL database to the view?
> > >         >
> > >         > Glib::ustring implicitly (and in my opinion still at least
> > >         not well
> > >         > enough documented/hinted at) converts from UTF-8
> > >         > to the current locale with operators << and >>. If you are
> > >         using a
> > >         > stringstream anywhere in the process, or are
> > >         > using operator<< or operator>> in any other way, this could
> > >         then be
> > >         > the cause of the problem.
> > >         >
> > >         > Otherwise, and in the above described case too actually it
> > >         would help
> > >         > very much if you could post (if possible, all of)
> > >         > the relevant code online.
> > >         >
> > >         > Regards,
> > >         > M.
> > >         >
> > >         > On Sun, Dec 28, 2008 at 8:53 PM, Bernd Robertz
> > >         > <bernd.robertz@...> wrote:
> > >         >         Hello list,
> > >         >
> > >         >         I'm not really sure, but I hope My question is in
> > >         the right
> > >         >         place here.
> > >         >
> > >         >         I wrote a simple database app like this:
> > >         >
> > >         >         Connect to a MySQL server via libmysql++,
> > >         >         and then list the rows in a treeview.
> > >         >         Doubleclick on a row shows a window with some
> > >         entries and a
> > >         >         textview
> > >         >         with the content of the selected row.
> > >         >         The textview is associated with a type text in the
> > >         DB.
> > >         >
> > >         >         Everything is fine.
> > >         >
> > >         >         So but I ran really into problems with the charsets.
> > >         >         german umlaute (ö, ä, etc.) and some other
> > >         characters are
> > >         >         glyphs in the
> > >         >         treeview. Ok, thats not the main problem.
> > >         >
> > >         >         That is: The entries in the detailed view still
> > >         shows the
> > >         >         glyphs for
> > >         >         some chars like ö, ä.
> > >         >         But on some rows, the represented textbuffer for
> > >         textview is
> > >         >         not
> > >         >         visible. Wich means the textview is in some rows
> > >         empty.
> > >         >
> > >         >         I run My app from a console and I got these error
> > >         messages:
> > >         >
> > >         >         :::::::::
> > >         >         Gtk-CRITICAL **: gtk_text_buffer_emit_insert:
> > >         assertion
> > >         >         `g_utf8_validate
> > >         >         (text, len, NULL)' failed
> > >         >         :::::::::
> > >         >
> > >         >         In other rows, the textbuffer is correct shown,
> > >         exepct some
> > >         >         glyphs.
> > >         >
> > >         >         If I print the textbuffer as a simple .c_str() in a
> > >         console it
> > >         >         will be
> > >         >         shown correctly (inklusive ä, ö, ü...).
> > >         >         (Tried to show the textview as a .c_str(), without
> > >         any
> > >         >         effect.)
> > >         >
> > >         >         The SQL connection is serverside-configured as UTF-8
> > >         Unicode.
> > >         >         The SQL table collation is UTF-8 General CI (Case
> > >         >         Insensitive).
> > >         >         The Application runs on a DE_de@Euro (ISO-8859-15).
> > >         >
> > >         >         All characters are shown correctly in a PHP app wich
> > >         does the
> > >         >         same, just
> > >         >         only online. :-)
> > >         >
> > >         >         I hope someone could turn on a light above my head,
> > >         why I
> > >         >         don't get the
> > >         >         right charactes.
> > >         >
> > >         >         Thanks alot in advance
> > >         >
> > >         >         Bernd
> > >         >
> > >         >
> > >         >         _______________________________________________
> > >         >         gtkmm-list mailing list
> > >         >         gtkmm-list@...
> > >         >         http://mail.gnome.org/mailman/listinfo/gtkmm-list
> > >         >
> > >         >
> > >         >
> > >         > --
> > >         > Please note that according to the German law on data
> > >         retention,
> > >         > information on every electronic information exchange with me
> > >         is
> > >         > retained for a period of six months.
> > >         > [Bitte beachten Sie, dass dem Gesetz zur
> > >         Vorratsdatenspeicherung
> > >         > zufolge
> > >         > jeder elektronische Kontakt mit mir sechs Monate lang
> > >         gespeichert
> > >         > wird.]
> > >        
> > >        
> > >
> > >
> > >
> > > --
> > > Please note that according to the German law on data retention,
> > > information on every electronic information exchange with me is
> > > retained for a period of six months.
> > > [Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung
> > > zufolge
> > > jeder elektronische Kontakt mit mir sechs Monate lang gespeichert
> > > wird.]
> >
> > _______________________________________________
> > gtkmm-list mailing list
> > gtkmm-list@...
> > http://mail.gnome.org/mailman/listinfo/gtkmm-list
>
> _______________________________________________
> gtkmm-list mailing list
> gtkmm-list@...
> http://mail.gnome.org/mailman/listinfo/gtkmm-list

_______________________________________________
gtkmm-list mailing list
gtkmm-list@...
http://mail.gnome.org/mailman/listinfo/gtkmm-list

[Bernd Robertz-2]

Hello,

well, first it was hard to figure out where the problem was.

I played around with different charsets on My Linux-Box, on the database
server and the webserver (with my PHP frontend for the same data).

After some (unsuccessful) attempts, I figured out, that everything was
ok on the client and on the server side, also with the conversion
from/to ustring<->strings in My Gtkmm application.

The solution was just be a conversion from html-entities wich are also
utf-8 characters to be valid c-lib utf-8 chars.

I simply used the multibyte-conversion function from php to convert the
data from my html-entities-utf-8 PHP-app to real utf-8 data and stored
them in My DB.

The embrassing thing is, that I should know, that http-form data are not
stored as the same character data, as it is in real applications.
(Never trust what your browser shows you. :-)) )
I think that have to do with the character-conversion-table in PHP.

So I just simply store and load them from My PHP-app as
multibyte-converted-strings. With this conversion, they are also
compatible to My Gtkmm application.

I'm not sure if this is a good way, but it works. :D

Thanks to your suggestions about the operand "<<" ">>" some posts ago.

If someone has a better or other way, feel free, to show Me another
point of view.

Best regards

Bernd


Am Samstag, den 03.01.2009, 15:59 +0100 schrieb Milosz Derezynski:

> How did you fix it? After all this back-and-forth it would be really
> interesting to know.
>
> Regards,
> M.
>
> On Sat, Jan 3, 2009 at 11:03 AM, Bernd Robertz
> <bernd.robertz@...> wrote:
>         Hello again,
>        
>         I fixed the problem.
>        
>         I know why I don't receive an answer about that. The solution
>         was so
>         easy that it was a bit embrassing. :-)
>        
>         Sorry for bothering and thanks to everyone who helped Me out.
>        
>         Greetings
>        
>         Bernd
>        
>        
>         Am Freitag, den 02.01.2009, 14:30 +0100 schrieb Bernd Robertz:
>        
>         > Hello all,
>         >
>         > I hope everyone is well arrived in the new year.
>         >
>         > I come back to my last post.
>         > It seems to be, that I have some weired missconfiguration
>         with my
>         > charsets.
>         > If I'm inserting a row from My gtkmm-app, it will be shown
>         correctly in
>         > My app, but not in other apps like my PHP webfrontend or
>         phpMyAdmin.
>         >
>         > I think I know how to fix it.
>         >
>         > Has anyone suggestions wich charset should generally be used
>         i. e. for
>         > client and server apps (Gtkmm, webserver, db, etc.)?
>         >
>         > Thanks everyone who give Me the flashlight to get the
>         problem.
>         >
>         > Many thanks
>         >
>         > Bernd
>         >
>         >
>         > Am Montag, den 29.12.2008, 13:09 +0100 schrieb Bernd
>         Robertz:
>         > > Hello,
>         > >
>         > > thanks for the code.
>         > >
>         > > result is an object of mysqlpp::StoreQueryResult in the
>         sql-libs.
>         > > It stores the query data.
>         > > But in the docs of libmysql++ is described, that are the
>         stored data
>         > > handeld as a derived std::string. Also compatible to the
>         original
>         > > std::string.
>         > >
>         > > I inserted your code as you described and it returns the
>         following
>         > > output (2 textviews as example, one with shown data, one
>         with empty)
>         > > The output is taken from console:
>         > >
>         > > :::::::::::::::::: SNIP first example
>         data :::::::::::::::::
>         > >
>         > > ** Message: UTF-8 valid: 0: Demozugang ist der gleiche.
>         > >
>         > > Auch f\xfcr gesch\xe4ftliche Aktivit\xe4ten (Streeting
>         > > Unternehmensvertrieb, siehe Unterlagen).
>         > >
>         > > Vertriebsnummer: 00028
>         > >
>         > > Vertriebsseite:
>         > > URL: http://vertrieb.streeting.com
>         > > Benutzer: xxxx
>         > > Kennwort: xxxx
>         > >
>         > > Abrechnungssystem (im Vertriebslogin):
>         > > Benutzer: xxxx
>         > > Kennwort: xxxx
>         > >
>         > > :::::::::::::::::: SNIP second example
>         data :::::::::::::::::
>         > >
>         > > ** Message: UTF-8 valid: 1: Daten CySoTec:
>         > > --------------
>         > >
>         > > Jabba (XMPP) Messenger Client:
>         > > Benutzer: xxxx
>         > > Kennwort: xxxx
>         > > Server: cysotec.info
>         > > Jabba ID: xrobbi@...
>         > > [...]
>         > >
>         > > :::::::::::::::::: SNIP example data end :::::::::::::::::
>         > >
>         > > As far as I can see, the one with empty data is not UTF-8
>         valid
>         > > and the other is.
>         > > So, what can I do, to convert the Data from the SQL server
>         to My app in
>         > > a valid UTF-8, or on the other hand,
>         > > what collation should I use in on My SQL server?
>         > >
>         > > Thank you alot
>         > >
>         > > Bernd
>         > >
>         > >
>         > > Am Montag, den 29.12.2008, 00:47 +0100 schrieb Milosz
>         Derezynski:
>         > > > Of what type is 'result'?
>         > > >
>         > > > Also, could you add, before this line:
>         > > >
>         > > >    tmp.push_back((std::string)result[row][i]);
>         > > >
>         > > > this code:
>         > > >
>         > > >    Glib::ustring utf8_str(result[row][i]);
>         > > >    g_message("UTF-8 valid: %d: %s",
>         int(utf8_str.validate()),
>         > > > utf8_str.c_str());
>         > > >
>         > > > and see what it gives?
>         > > >
>         > > >
>         > > > M.
>         > > >
>         > > > P.S.:
>         > > > Yes std::string and ustring are generally compatible,
>         with the main
>         > > > differences being that ustring works by-character not
>         by-byte (e.g.
>         > > > find(), operator[], etc), ustring has methods for
>         validation of UTF-8
>         > > > and related functionality; the main potential culprit
>         here is
>         > > > operator<< and operator>> of ustring. I had really big
>         problems with
>         > > > these using boost::format() until i realised that
>         internally, it's a
>         > > > stream to which the args are fed using operator<<() of
>         ustring.
>         > > >
>         > > >
>         > > > On Mon, Dec 29, 2008 at 12:01 AM, Bernd Robertz
>         > > > <bernd.robertz@...> wrote:
>         > > >         Thanks alot for your fast reply,
>         > > >
>         > > >         I wrote a simple MySQL-Class wich is derived
>         from the libsql++
>         > > >         class.
>         > > >
>         > > >         In this class I used std::string to get the
>         columns of a row.
>         > > >         The columns (strings) are stored in a
>         vector<string> wich I
>         > > >         return to
>         > > >         Glib::ustring in My treeview-data-window.
>         > > >
>         > > >         Hiere are some snippets:
>         > > >
>         > > >         ::::::::::::::::::::::::
>         SNIP ::::::::::::::::::::::
>         > > >         database.cpp
>         > > >         (My simple MySQL database class.)
>         > > >
>         > > >         DataBase::Database()
>         > > >         {
>         > > >                // db-connect
>         > > >                // query-setup
>         > > >                // other connect related stuff
>         > > >
>         > > >                mysqlpp::StoreQueryResult result; //
>         Setup the result
>         > > >         object
>         > > >                int rows; // Keep track of result counts
>         > > >         }
>         > > >
>         > > >         void DataBase::returnrow(int dataid)
>         > > >         {
>         > > >                // Setup db-query with dataid to get
>         requested db-row
>         > > >                //
>         > > >                mysqlpp::Query query =
>         dbcon.query(queries.alldata); //
>         > > >         DB Query
>         > > >                result = query.store(); // Store the
>         results
>         > > >                rows = result.num_rows(); // Set the
>         number of results
>         > > >         }
>         > > >
>         > > >         std::vector<std::string> DataBase::fetchdata(int
>         row)
>         > > >         {
>         > > >                std::vector<std::string> tmp; // Temp
>         vector for return
>         > > >
>         > > >                for (int i = 0; i <
>         result.field_names()->size(); ++i)
>         > > >                {
>         > > >                        // Every column will be stored in
>         the temp
>         > > >         vector
>         > > >
>          tmp.push_back((std::string)result[row][i]);
>         > > >                }
>         > > >                return tmp; // Return the temp-vector
>         with the
>         > > >         [columns]=data
>         > > >         }
>         > > >
>         > > >         ------------
>         > > >
>         > > >         datawindow.cpp
>         > > >         (The window in where the data is displayed after
>         dbl-click on
>         > > >         treeview)
>         > > >
>         > > >         DataWindow::DataWindow()
>         > > >         {
>         > > >                // Widgets setup
>         > > >                // Database connect via DataBase-Object
>         > > >         }
>         > > >
>         > > >         void DataWindow::getid(int dataid)
>         > > >         {
>         > > >                // From the treeview I call this method
>         to set the
>         > > >                // requested row-id
>         > > >                mysql.returnrow(dataid);
>         > > >
>         > > >                // Call this method to setup the entries
>         and textview
>         > > >                setdata();
>         > > >         }
>         > > >
>         > > >         void DataWindow::setdata()
>         > > >         {
>         > > >                l_dataid.set_text(mysql.fetchdata(0)[0]);
>         > > >                Gtk::TreeModel::Row selrow =
>         *(ls_cbox->prepend());
>         > > >                selrow[c_mod.c_cat] =
>         mysql.fetchdata(0)[1];
>         > > >                cbox.set_active(0);
>         > > >
>         > > >                // Here comes the problem, I get every
>         column from My
>         > > >                // database-object wich I store in the
>         entries.
>         > > >                e_name.set_text(mysql.fetchdata(0)[2]);
>         > > >                e_number.set_text(mysql.fetchdata(0)[3]);
>         > > >                e_user.set_text(mysql.fetchdata(0)[4]);
>         > > >                e_pass.set_text(mysql.fetchdata(0)[5]);
>         > > >                e_mail.set_text(mysql.fetchdata(0)[6]);
>         > > >                e_url.set_text(mysql.fetchdata(0)[7]);
>         > > >                e_other.set_text(mysql.fetchdata(0)[8]);
>         > > >
>         > > >                // Here we have the main problem: the
>         textview-buffer
>         > > >                // Some rows are shown here, some not.
>         > > >                b_descr->set_text(mysql.fetchdata(0)[9]);
>         > > >         }
>         > > >
>         > > >         ::::::::::::::::::::::::: SNIP
>         > > >         END ::::::::::::::::::::::::::::
>         > > >
>         > > >         That is the main code where the problem occours.
>         > > >
>         > > >         The DataWindow is called on doubleclick on a
>         treeview-row.
>         > > >         With the call of the DataWindow I call the
>         DataView::getid()
>         > > >         wich receive the database row-id.
>         > > >         Then I fill with the DataView::setdata() method
>         the widgets
>         > > >         in the new window.
>         > > >
>         > > >         Some chars are, as I told, glyphs, but the
>         textbuffer is the
>         > > >         main
>         > > >         problem.
>         > > >
>         > > >         As far as I know, the entries and textbuffers
>         are expecting
>         > > >         Glib::ustring, but I thought the conversion from
>         std::string
>         > > >         to
>         > > >         Glib::ustring is quite compatible between these
>         types.
>         > > >         I used std::string in the database class,
>         because I wanted to
>         > > >         isolated
>         > > >         it from Gtkmm for other not Gtkmm-related
>         projects.
>         > > >
>         > > >         Thank you for helping.
>         > > >
>         > > >         Bernd
>         > > >
>         > > >
>         > > >         Am Sonntag, den 28.12.2008, 22:26 +0100 schrieb
>         Milosz
>         > > >         Derezynski:
>         > > >
>         > > >         > Are you using Glib::ustring anywhere in the
>         process of
>         > > >         getting the
>         > > >         > data from the SQL database to the view?
>         > > >         >
>         > > >         > Glib::ustring implicitly (and in my opinion
>         still at least
>         > > >         not well
>         > > >         > enough documented/hinted at) converts from
>         UTF-8
>         > > >         > to the current locale with operators << and
>         >>. If you are
>         > > >         using a
>         > > >         > stringstream anywhere in the process, or are
>         > > >         > using operator<< or operator>> in any other
>         way, this could
>         > > >         then be
>         > > >         > the cause of the problem.
>         > > >         >
>         > > >         > Otherwise, and in the above described case too
>         actually it
>         > > >         would help
>         > > >         > very much if you could post (if possible, all
>         of)
>         > > >         > the relevant code online.
>         > > >         >
>         > > >         > Regards,
>         > > >         > M.
>         > > >         >
>         > > >         > On Sun, Dec 28, 2008 at 8:53 PM, Bernd Robertz
>         > > >         > <bernd.robertz@...> wrote:
>         > > >         >         Hello list,
>         > > >         >
>         > > >         >         I'm not really sure, but I hope My
>         question is in
>         > > >         the right
>         > > >         >         place here.
>         > > >         >
>         > > >         >         I wrote a simple database app like
>         this:
>         > > >         >
>         > > >         >         Connect to a MySQL server via libmysql
>         ++,
>         > > >         >         and then list the rows in a treeview.
>         > > >         >         Doubleclick on a row shows a window
>         with some
>         > > >         entries and a
>         > > >         >         textview
>         > > >         >         with the content of the selected row.
>         > > >         >         The textview is associated with a type
>         text in the
>         > > >         DB.
>         > > >         >
>         > > >         >         Everything is fine.
>         > > >         >
>         > > >         >         So but I ran really into problems with
>         the charsets.
>         > > >         >         german umlaute (ö, ä, etc.) and some
>         other
>         > > >         characters are
>         > > >         >         glyphs in the
>         > > >         >         treeview. Ok, thats not the main
>         problem.
>         > > >         >
>         > > >         >         That is: The entries in the detailed
>         view still
>         > > >         shows the
>         > > >         >         glyphs for
>         > > >         >         some chars like ö, ä.
>         > > >         >         But on some rows, the represented
>         textbuffer for
>         > > >         textview is
>         > > >         >         not
>         > > >         >         visible. Wich means the textview is in
>         some rows
>         > > >         empty.
>         > > >         >
>         > > >         >         I run My app from a console and I got
>         these error
>         > > >         messages:
>         > > >         >
>         > > >         >         :::::::::
>         > > >         >         Gtk-CRITICAL **:
>         gtk_text_buffer_emit_insert:
>         > > >         assertion
>         > > >         >         `g_utf8_validate
>         > > >         >         (text, len, NULL)' failed
>         > > >         >         :::::::::
>         > > >         >
>         > > >         >         In other rows, the textbuffer is
>         correct shown,
>         > > >         exepct some
>         > > >         >         glyphs.
>         > > >         >
>         > > >         >         If I print the textbuffer as a
>         simple .c_str() in a
>         > > >         console it
>         > > >         >         will be
>         > > >         >         shown correctly (inklusive ä, ö,
>         ü...).
>         > > >         >         (Tried to show the textview as
>         a .c_str(), without
>         > > >         any
>         > > >         >         effect.)
>         > > >         >
>         > > >         >         The SQL connection is
>         serverside-configured as UTF-8
>         > > >         Unicode.
>         > > >         >         The SQL table collation is UTF-8
>         General CI (Case
>         > > >         >         Insensitive).
>         > > >         >         The Application runs on a DE_de@Euro
>         (ISO-8859-15).
>         > > >         >
>         > > >         >         All characters are shown correctly in
>         a PHP app wich
>         > > >         does the
>         > > >         >         same, just
>         > > >         >         only online. :-)
>         > > >         >
>         > > >         >         I hope someone could turn on a light
>         above my head,
>         > > >         why I
>         > > >         >         don't get the
>         > > >         >         right charactes.
>         > > >         >
>         > > >         >         Thanks alot in advance
>         > > >         >
>         > > >         >         Bernd
>         > > >         >
>         > > >         >
>         > > >         >
>         _______________________________________________
>         > > >         >         gtkmm-list mailing list
>         > > >         >         gtkmm-list@...
>         > > >         >
>         http://mail.gnome.org/mailman/listinfo/gtkmm-list
>         > > >         >
>         > > >         >
>         > > >         >
>         > > >         > --
>         > > >         > Please note that according to the German law
>         on data
>         > > >         retention,
>         > > >         > information on every electronic information
>         exchange with me
>         > > >         is
>         > > >         > retained for a period of six months.
>         > > >         > [Bitte beachten Sie, dass dem Gesetz zur
>         > > >         Vorratsdatenspeicherung
>         > > >         > zufolge
>         > > >         > jeder elektronische Kontakt mit mir sechs
>         Monate lang
>         > > >         gespeichert
>         > > >         > wird.]
>         > > >
>         > > >
>         > > >
>         > > >
>         > > >
>         > > > --
>         > > > Please note that according to the German law on data
>         retention,
>         > > > information on every electronic information exchange
>         with me is
>         > > > retained for a period of six months.
>         > > > [Bitte beachten Sie, dass dem Gesetz zur
>         Vorratsdatenspeicherung
>         > > > zufolge
>         > > > jeder elektronische Kontakt mit mir sechs Monate lang
>         gespeichert
>         > > > wird.]
>         > >
>         > > _______________________________________________
>         > > gtkmm-list mailing list
>         > > gtkmm-list@...
>         > > http://mail.gnome.org/mailman/listinfo/gtkmm-list
>         >
>         > _______________________________________________
>         > gtkmm-list mailing list
>         > gtkmm-list@...
>         > http://mail.gnome.org/mailman/listinfo/gtkmm-list
>        
>         _______________________________________________
>         gtkmm-list mailing list
>         gtkmm-list@...
>         http://mail.gnome.org/mailman/listinfo/gtkmm-list
>        
>
>
>
> --
> Please note that according to the German law on data retention,
> information on every electronic information exchange with me is
> retained for a period of six months.
> [Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung
> zufolge
> jeder elektronische Kontakt mit mir sechs Monate lang gespeichert
> wird.]

_______________________________________________
gtkmm-list mailing list
gtkmm-list@...
http://mail.gnome.org/mailman/listinfo/gtkmm-list