[issue4087] equality involving Decimals is not transitive; strange set behaviour results

[Martin v. Löwis-2]

New submission from Mark Dickinson <dickinsm@...>:

The Decimal module breaks transitivity of equality:  Decimal(2) == 2 and
2 == float(2), but Decimal(2) != float(2).

The Python set and dict implementations rely on transitivity of equality
for correct operation.

These two facts together give some strange results when playing with
sets and dicts involving Decimals and floats.  For example (with Python
2.6):

>>> s = set([Decimal(2), float(2)])
>>> t = set([2])
>>> s | t == t | s
False
>>> len(s | t)
2
>>> len(t | s)
1

Other strange examples, and possible solutions, were discussed recently
on comp.lang.python;  see the thread starting at:

http://mail.python.org/pipermail/python-list/2008-September/508859.html

Possible solutions:

(1) Document the problem, making it clear in the language reference that
correct set operation relies on transitivity of equality, and adding a
note to the decimal documentation indicating that mixing floats and
Decimals in a container is asking for trouble.

(2) Fix up Decimal so that equal numeric objects compare equal; this
would also involve fixing the hash operation.  To me, this goes against
the philosophy of keeping the Decimal module close to the specification.

(3) Terry Reedy suggested (in the c.l.python thread linked to above) a
simpler version of (2): allow Decimal(i) == float(i) or Decimal(i) ==
Fraction(i) to return True for all integers i.  (Decimal('0.5') == 0.5
would still return False.)  This fixes transitivity, and has the
advantage of not requiring any changes to the hash functions:  
hash(Decimal(i)) == hash(float(i)) is already true for all integers i.

My own preference would be simply to document the problem;  it doesn't
seem like something that's going to affect that vast majority of Python
users.

Raymond, Facundo:  any thoughts?

----------
messages: 74576
nosy: facundobatista, marketdickinson, rhettinger, tjreedy
priority: normal
severity: normal
status: open
title: equality involving Decimals is not transitive;  strange set behaviour results
type: behavior
versions: Python 2.6, Python 2.7, Python 3.0

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4087>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[Martin v. Löwis-2]

Terry J. Reedy <tjreedy@...> added the comment:

There are two issues involved:
1. documenting set behavior
2. what to do, if anything, about Decimals and other numbers

Since users are free to create similar problems, and since sets are
missing from the Reference section on comparisons, I opened a separate
set documentation issue http://bugs.python.org/issue4090
leaving this as a Decimal-other_number equality issue.

The root of the problem is that all members of s are members of t and
vice versa.  This should make s and t equal, but they are not.  This
also breaks the definitions of issubset (<=), issuperset (>=), union
(|), and symmetric_difference (^) as shown in the c.l.p thread.

Transitivity is also fundamental in logic and the rule of substitution.
 So I strongly prefer that it be preserved in Python as released.

Another way to restore transitivity is
(4) make Decimal(1) != int(1) just as Decimal(1) != float(1).  Having a
Decimal be equal in value to just one of two things that are equal in
value is incoherent, and leads to incoherent results.

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4087>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[Martin v. Löwis-2]

Facundo Batista <facundo@...> added the comment:

(Ok, remember that I'm not a "numeric" guy before start hitting me, :p )

I think that if we have Decimal(1)==1, and 1==1.0, to have Decimal(1)==1.0.

We always rejected comparison with "unsupported types", but having this
situation, I'd propose to put something like the following at the
beggining of __eq__() and similars:

    def __eq__(self, other):
        if isinstance(other, float) and int(other)==other:
            other = int(other)

What do you think?

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4087>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[Martin v. Löwis-2]

Raymond Hettinger <rhettinger@...> added the comment:

Recommend not "doing anything" about decimals and other numbers.  What
you're seeing is a predictable consequence of NotImplemented being
returned by some but not all cross type comparisons.  IMO, it is
perfectly reasonable that both decimals and floats can be compared to
integers but not to each other.  Integers are a "universal donor" in
this respect but the two float types are not.

It is true that equality should be transitive but the same cannot be
said for the *ability of types* to be compared.  Unfortunately, the ==
operator masks what is going on by returning False instead of raising a
NotImplementedError.  IOW, it the apparant loss of transitivity when
"float(2) == Decimal(2)" returns False is an illusion; instead, the
False return means that the types cannot be compared at all.

If any doc changes are made with respect to this issue, it should be in
the docs for the == and != operators and for NotImplemented.

----------
assignee:  -> georg.brandl
components: +Documentation
nosy: +georg.brandl
priority: normal -> low
title: equality involving Decimals is not transitive;  strange set behaviour results -> Document the effects of NotImplemented on == and !=

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4087>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[Martin v. Löwis-2]

Terry J. Reedy <tjreedy@...> added the comment:

If Decimal(2) == float(2) were to raise an error,
set([Decimal(2), float(2)]) would fail, as I would argue it ought to,
and the set anomalies would disappear.

----------
assignee: georg.brandl ->
priority: low -> normal

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4087>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[Martin v. Löwis-2]

Raymond Hettinger <rhettinger@...> added the comment:

I'll take up the doc fix for the current state of affairs.

A change from returning NotImplemented to raising NotImplementedError
would need be a separate feature request or PEP.

Alternatively, we could decide to allow decimal/float comparisons -- the
float can be converted to a decimal exactly and compared exactly -- it
would be slow but it would work and have precise semantics.

Am resetting the priority back to low because the current behavior is
exactly what is supposed to happen.  It is an automatic consequence of
how we use NotImplemented and the decision to allow integer/float and
integer/decimal comparisons but not float/decimal comparisons.  There is
not bug here, just something that offends ones sensibilities.

----------
assignee:  -> rhettinger
priority: normal -> low

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4087>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[Martin v. Löwis-2]

Facundo Batista <facundo@...> added the comment:

2008/10/9 Raymond Hettinger <report@...>:

> Alternatively, we could decide to allow decimal/float
> comparisons -- the float can be converted to a decimal
> exactly and compared exactly -- it would be slow but
> it would work and have precise semantics.

-0

Note that this could lead to surprising behaviours, when doing these
comparations... Decimal("1.1")==1.1 will be true, buy maybe
Decimal("1.235445687")==1.235445687 will not (I didn't try if this
particular comparison will fail, but hope you get the idea).

This is why I suggested the other way... we now allow comparison to
integers, let's allow comparisons when the floats are equal to the
integers, and no more.

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4087>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[Martin v. Löwis-2]

Terry J. Reedy <tjreedy@...> added the comment:

More sensibility offenders:

>>> s = {fractions.Fraction(17,1), decimal.Decimal(17)}
>>> s-{17}
set()

Removing one thing removes two.

>>> s.remove(17)
>>> 17 in s
True

Removing something leaves it there.

>>> s
{Fraction(17, 1)} # random choice

Removing one thing and subtracting the set with that one thing give
different results.

>>> s = {decimal.Decimal(17), fractions.Fraction(17,1)}
>>> s.remove(17)
>>> s
{Decimal('17')}

The behavior of 'set' s depends on the order items are added.

> Facundo's suggested code:

    if isinstance(other, float) and int(other)==other:
        other = int(other)

would be more efficient, I assume as

    if isinstance(other,float):
        ifloat = int(other)
        if other == ifloat:
            other = ifloat

or if the CAPI has an efficient 'float_isint' function that accesses the
bits of a float, as the C equivalent of

    if isinstance(other, float) and float_isint(other):
        other = int(other)

I remember float-Decimal comparison being rejected/deferred in part for
being problematical for fractional values.  That is why I suggested
implementing it, at least for the present, for integral floats (and
Fractions) only, which are relatively easy to detect.

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4087>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[Martin v. Löwis-2]

Raymond Hettinger <rhettinger@...> added the comment:

r67249

----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4087>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com