Multiple ETS lookups

[Colm Dougan]

Hi list,

What do people think of the idea of adding a multiple ETS lookup
function, analogous to the SQL 'IN' operator, where you can pass a
list of keys and you get a list of matches?

Something like :

1> T0 = ets:new(testtable, [set]).
15
2> ets:insert(T0, [{foo, 10}, {bar, 20}, {baz, 30}]).
true
3> ets:multi_lookup(T0, [foo, bar, baz]).
[{foo, 10},{bar,20},{baz,30}]
4> ets:multi_lookup(T0, [dog, foo, cat]).
[{foo,10}]

My motivation for this suggestion is mainly performance but I think it
is also a common enough idiom that the interface would be generally
useful.

I realize that this problem has been solved in a more general way with
match specs but unless I'm missing something using a match spec (even
a compiled one) to do (say) 3 lookups is slower than doing 3
individual ets:lookup calls which defeats the point (in that case).
Am I missing something here?  I also find the match spec interface a
bit unwieldy for such a simple operation (although I realize it is
designed for flexibility).

In my case the multiple ets lookup would help work around the
performance dip I got in critical sections doing intensive lookups
when I moved from non-SMP mode to SMP mode.  I realize that relatively
speaking the SMP locking is not that expensive unless there are lock
conflicts but there is still a cost and when doing intensive lookups
you definitely notice it.   Being able to "unroll" my lookups to do
(say) 4 or 8 at once and only incur the locking overheads a quarter of
the time would be a big win.

Yeah I know people will then potentially be able to do nasty things
like try to do too many lookups at once, but then again they could
just not do that.

Anyway, just throwing the idea out there to be shot at.

Thanks,
Colm
_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Ulf Wiger]

I posted a slightly more general suggestion a while back:

http://www.erlang.org/pipermail/erlang-questions/2007-May/026440.html
http://www.erlang.org/pipermail/erlang-questions/2007-June/026998.html
http://www.erlang.org/pipermail/erlang-questions/2007-June/027004.html

The main reason for doing this, I think, would be atomicity, without
having to go either through a serializing process or mnesia.

Conceptually, using a process is a lot cleaner. The problem
(apart from forcing you to write more code) is that you double the
amount of copying needed...

BR,
Ulf W

2008/10/1 Colm Dougan <colm.dougan@...>:

> Hi list,
>
> What do people think of the idea of adding a multiple ETS lookup
> function, analogous to the SQL 'IN' operator, where you can pass a
> list of keys and you get a list of matches?
>
> Something like :
>
> 1> T0 = ets:new(testtable, [set]).
> 15
> 2> ets:insert(T0, [{foo, 10}, {bar, 20}, {baz, 30}]).
> true
> 3> ets:multi_lookup(T0, [foo, bar, baz]).
> [{foo, 10},{bar,20},{baz,30}]
> 4> ets:multi_lookup(T0, [dog, foo, cat]).
> [{foo,10}]
>
> My motivation for this suggestion is mainly performance but I think it
> is also a common enough idiom that the interface would be generally
> useful.
>
> I realize that this problem has been solved in a more general way with
> match specs but unless I'm missing something using a match spec (even
> a compiled one) to do (say) 3 lookups is slower than doing 3
> individual ets:lookup calls which defeats the point (in that case).
> Am I missing something here?  I also find the match spec interface a
> bit unwieldy for such a simple operation (although I realize it is
> designed for flexibility).
>
> In my case the multiple ets lookup would help work around the
> performance dip I got in critical sections doing intensive lookups
> when I moved from non-SMP mode to SMP mode.  I realize that relatively
> speaking the SMP locking is not that expensive unless there are lock
> conflicts but there is still a cost and when doing intensive lookups
> you definitely notice it.   Being able to "unroll" my lookups to do
> (say) 4 or 8 at once and only incur the locking overheads a quarter of
> the time would be a big win.
>
> Yeah I know people will then potentially be able to do nasty things
> like try to do too many lookups at once, but then again they could
> just not do that.
>
> Anyway, just throwing the idea out there to be shot at.
>
> Thanks,
> Colm
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@...
> http://www.erlang.org/mailman/listinfo/erlang-questions
>

_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions

[Colm Dougan]

> 2008/10/1 Colm Dougan <colm.dougan@...>:
>> Hi list,
>>
>> What do people think of the idea of adding a multiple ETS lookup
>> function, analogous to the SQL 'IN' operator, where you can pass a
>> list of keys and you get a list of matches?

On Wed, Oct 1, 2008 at 7:26 AM, Ulf Wiger <ulf@...> wrote:
> I posted a slightly more general suggestion a while back:
>
> http://www.erlang.org/pipermail/erlang-questions/2007-May/026440.html

That looks really useful.  IMO there is room for both functions (seq
and multi_lookup). In the first instance multi_lookup could be
implemented in C but at a later date if your much more powerful seq
stuff gets added then that may enable multi_lookup to be implemented
in terms of seq.

I'm happy to have a stab at putting together a patch for multi_lookup
if there is any interest?  I'm keen to know if people like the idea,
don't like the idea or are indifferent to it :)

Colm
_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions