Hebrew under Unbuntu

[Ilan]

I wrote a desktop application under Windows which uses Hebrew.
When I run it under Ubuntu the Hebrew comes up just fine (run the jar file).
I wanted to change something, so I wanted to move the Netbeans project over to Ubuntu.

Under Ubuntu OpenOffice works fine with Hebrew. I can even see Hebrew characters in the terminal.
Netbeans, for some strange reason, won't accept Hebrew characters.
I try to drop in a push button and change the text to Hebrew.
Depending upon if I do it in Properties, or a click on the button it will give me either English characters, or just boxes where the characters should be.

Presumably I have to "turn on" something in Netbeans to accept Hebrew, but I don't know what.

Thanks,
Ilan

[kenf]

Ilan,

it might be about how java is  mapping the fonts and perhaps that
ubuntu jdk is needed if that one is not being used (vs for example the
jdk from java.sun.com) (since the square boxes show this might be happening)

first, if you run a jdk demo program, does the Hebrew show ok ?
if not, then it could be the jdk.

please see this faq page for info on how to get and setup the ubuntu jdk.

http://wiki.netbeans.org/FaqI18nLinuxAsianFonts

and sections in it on Ubuntu 7.x and 6.x

Thanks - Ken

Ilan wrote:

I wrote a desktop application under Windows which uses Hebrew.
When I run it under Ubuntu the Hebrew comes up just fine (run the jar file).
I wanted to change something, so I wanted to move the Netbeans project over to Ubuntu.

Under Ubuntu OpenOffice works fine with Hebrew. I can even see Hebrew characters in the terminal.
Netbeans, for some strange reason, won't accept Hebrew characters.
I try to drop in a push button and change the text to Hebrew.
Depending upon if I do it in Properties, or a click on the button it will give me either English characters, or just boxes where the characters should be.

Presumably I have to "turn on" something in Netbeans to accept Hebrew, but I don't know what.

Thanks,
Ilan

[Ilan]

Ken,
Thanks for your VERY GOOD idea. I can't find the jdk demo program so I can't check it for Hebrew.
However your idea is very logical and probably correct.

I made a new project, a desktop application with a single button with a Hebrew caption.
I could see only boxes under Netbeans but when I ran the project, the Hebrew came up OK.
This could be explained by saying Netbeans is using the jdk and the application is running under jre.
The jre supports Hebrew and the jdk doesn't. (maybe?)

I see Netbeans is using Open source jdk, so I tried to install Sun's jdk.
I didn't see any improvement. Maybe I have to uninstall the default Open source of Ubuntu?

I'm having a similar problem with MySQL. It may be related or it may be a separate issue.

Thanks,
Ilan

[Ilan]

Ken,
Thanks as your idea worked. I replaced the Ubuntu jdk by the Sun jdk.
I had to go to /usr/share/netbeans/6.0.1/etc to find netbeans.conf and to change
it to point to the Sun jdk. After I reopened Netbeans I could suddenly see Hebrew!

So you were right on. It just took some additional digging to figure out how to configure it.
Thanks,
Ilan

[kenf]

Ilan,

I'm glad it worked.  Just to confirm, the sun jdk you installed was the one from
the ubuntu repository, not java.sun.com, since I think the one from ubuntu is customized
for font finding/naming, at least for asian locales.

for mysql, please let me know more details, and if you are using the netbeans/mysql bundle installer
or separate mysql installation.

I think for mysql, dbase and/or server connections need to have hebrew or perhaps utf-8 encoding
specified since I think default encoding is a latin one.

these pages might help
http://dev.mysql.com/doc/refman/5.1/en/internationalization-localization.html

in any case, please let me know details about use of netbeans connected to mysql and
if not seeing Hebrew ok in sql viewer or in your nb projects that interface/communicate with
your mysql.

Thanks - Ken

Ilan wrote:

Ken,
Thanks as your idea worked. I replaced the Ubuntu jdk by the Sun jdk.
I had to go to /usr/share/netbeans/6.0.1/etc to find netbeans.conf and to change
it to point to the Sun jdk. After I reopened Netbeans I could suddenly see Hebrew!

So you were right on. It just took some additional digging to figure out how to configure it.
Thanks,
Ilan

[Ilan]

Hi Ken,
You have been EXTREMELY helpful to getting me going on this.
First of all, I try to stay with Synaptic for installing packages.
Aside from plugins for Netbeans, as far as I remember, it has been enough to get the job done.
Thus I picked up the jdk from Synaptic, but it wasn't the default jdk which Ubuntu gave me.
The default Open-JDK (or something like that) didn't work for Hebrew. The Sun JDK works nicely.

Now for today's problem. I read somewhere along the line that I have to recompile MySQL sources in order to get Hebrew as a default language. I was dreading that since Windows has a nice utility from MySQL which allows me to change the default character set to Hebrew. After I did that, my problems in Windows were gone. I didn't know why Ubuntu couldn't do the same thing.

You pointed me to the MySQL documentation which I read in detail. There it mentions that you can configure the MySQL. This is exactly what I wanted to avoid compiling sources. It looked very promising but so far it won't fly. When I try to insert a Hebrew string, it tells me that it is illegal. Blah....

Let me show you the Windows version:

# MySQL Server Instance Configuration File
# ----------------------------------------------------------------------
# Generated by the MySQL Server Instance Configuration Wizard
#
#
# Installation Instructions
# ----------------------------------------------------------------------
#
# On Linux you can copy this file to /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options
# (@localstatedir@ for this installation) or to
# ~/.my.cnf to set user-specific options.
#
# On Windows you should keep this file in the installation directory
# of your server (e.g. C:\Program Files\MySQL\MySQL Server X.Y). To
# make sure the server reads the config file use the startup option
# "--defaults-file".
#
# To run run the server from the command line, execute this in a
# command line shell, e.g.
# mysqld --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
#
# To install the server as a Windows service manually, execute this in a
# command line shell, e.g.
# mysqld --install MySQLXY --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
#
# And then execute this in a command line shell to start the server, e.g.
# net start MySQLXY
#
#
# Guildlines for editing this file
# ----------------------------------------------------------------------
#
# In this file, you can use all long options that the program supports.
# If you want to know the options a program supports, start the program
# with the "--help" option.
#
# More detailed information about the individual options can also be
# found in the manual.
#
#
# CLIENT SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by MySQL client applications.
# Note that only client applications shipped by MySQL are guaranteed
# to read this section. If you want your own MySQL client program to
# honor these values, you need to specify it as an option during the
# MySQL client library initialization.
#
[client]

port=3306

[mysql]

default-character-set=hebrew


# SERVER SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by the MySQL Server. Make sure that
# you have installed the server correctly (see above) so it reads this
# file.
#
[mysqld]

# The TCP/IP Port the MySQL Server will listen on
port=3306


#Path to installation directory. All paths are usually resolved relative to this.
basedir="C:/Program Files/MySQL/MySQL Server 5.0/"

#Path to the database root
datadir="C:/Program Files/MySQL/MySQL Server 5.0/Data/"

# The default character set that will be used when a new schema or table is
# created and no character set is defined
default-character-set=hebrew

# The default storage engine that will be used when create new tables when
default-storage-engine=INNODB

# Set the SQL mode to strict
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections=100

# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.
query_cache_size=0

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_cache=256

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=9M


# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=8

#*** MyISAM Specific options

# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size=100G

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method.  This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.
myisam_max_extra_sort_file_size=100G

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method.  This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.
myisam_sort_buffer_size=18M

# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size=11M

# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=64K
read_rnd_buffer_size=256K

# This buffer is allocated when MySQL needs to rebuild the index in
# REPAIR, OPTIMZE, ALTER table statements as well as in LOAD DATA INFILE
# into an empty table. It is allocated per thread so be careful with
# large settings.
sort_buffer_size=256K


#*** INNODB Specific options ***


# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
#skip-innodb

# Additional memory pool that is used by InnoDB to store metadata
# information.  If InnoDB requires more memory for this purpose it will
# start to allocate it from the OS.  As this is fast enough on most
# recent operating systems, you normally do not need to change this
# value. SHOW INNODB STATUS will display the current amount used.
innodb_additional_mem_pool_size=2M

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit=1

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
innodb_log_buffer_size=1M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size=18M

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size=10M

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=8



I think the key is: default-character-set=hebrew

Thus I tried to put this into the Ubuntu version:


#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
#
# * Basic Settings
#

#
# * IMPORTANT
#   If you make changes to these settings and your system uses apparmor, you may
#   also need to also adjust /etc/apparmor.d/usr.sbin.mysqld.
#

user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/english
skip-external-locking
default-character-set=hebrew
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 128K
thread_cache_size = 8
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit       = 1M
query_cache_size        = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
#log = /var/log/mysql/mysql.log
#
# Error logging goes to syslog. This is a Debian improvement :)
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size         = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * BerkeleyDB
#
# Using BerkeleyDB is now discouraged as its support will cease in 5.1.12.
skip-bdb
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
# You might want to disable InnoDB to shrink the mysqld process by circa 100MB.
#skip-innodb
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem



[mysqldump]
quick
quote-names
max_allowed_packet = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
default-character-set=hebrew

[isamchk]
key_buffer = 16M

#
# * NDB Cluster
#
# See /usr/share/doc/mysql-server-*/README.Debian for more information.
#
# The following configuration is read by the NDB Data Nodes (ndbd processes)
# not from the NDB Management Nodes (ndb_mgmd processes).
#
# [MYSQL_CLUSTER]
# ndb-connectstring=127.0.0.1


#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/



As you can see, I added the Hebrew characterset, but it didn't help. I see from the above that I could have placed them in the conf.d directory, which is currently empty.
I would have expected the Windows and Linux versions would have the same commands, so my set default characterset should have worked. I don't understand why it doesn't do the job.

I'll continue to play with it, as maybe something will teach it what Hebrew is.

Thanks again for your help.
Ilan

[Ilan]

The problem is solved!

First of all I would like to express my deep appreciation to Ken Frank of Sun with whom I exchanged many private emails until we found the solution. He investigated the problem independently and gave me suggestions to try. In the end his most useful suggestion turned out to be the sql command:
show variables like 'character_set%'
which allowed me to see what was going on inside the database, rather than guess at it.

The solution is to set the database server to run in Hebrew. Everything else inherits from the database server so there is no need to explicitly specify the database or the tables be Hebrew - they will be by default.

All this can be done with a single line of code in my.cnf.
Open a terminal and go to the directory

cd /etc/mysql

In order to be able the modify the file use

sudo gedit my.cnf

Add the line:

[mysqld]
#
# * Basic Settings
#

#
# * IMPORTANT
#   If you make changes to these settings and your system uses apparmor, you may
#   also need to also adjust /etc/apparmor.d/usr.sbin.mysqld.
#
default-character-set = hebrew
user = mysql

You can see the line I added above as default-character-set = hebrew
which I placed in the server section.
Save the file and restart the server (I rebooted the system to restart the server).

Then you can create your database which will inherit the Hebrew from the server.
I then used NetBeans and connected to the database and ran the sql command
show variables like 'character_set%'
and got back the result:
character_set_client hebrew
character_set_connection hebrew
character_set_database hebrew
character_set_filesystem binary
character_set_results
character_set_server hebrew
character_set_system utf8
character_sets_dir /usr/share/mysql/charsets/

The most important line is character_set_server which is now Hebrew.
Also note that the database was created in Hebrew.

From here on it is a piece of cake. Everything works correctly including Open Office.

Ilan