Re: BTS not searching for numbers?

November 19, 2014, 9:55 am

≫ Next: Re: BTS not searching for numbers?

≪ Previous: BTS not searching for numbers?

$

0

0

I would guess you are not using the correct analyzer for you desired use
case.
The analyzer breaks the words into individual tokens to be searched.

To enable a different analyzer you can do the following;

create index desc_idx ON products ( brands bts_char_ops)
USING bts ( analyzer="whitespace") IN sbsp1;

John F. Miller III
STSM, Lead Architect
miller3@us.ibm.com
503-747-1366
IBM Informix Dynamic Server (IDS)

datablade-list-bounces@iiug.org wrote on 11/19/2014 07:48:16 AM:

> From: "MICHAEL HOFFMAN"<mrh@panix.com>
> To: datablade-list@iiug.org
> Date: 11/19/2014 07:48 AM
> Subject: BTS not searching for numbers? [336]
> Sent by: datablade-list-bounces@iiug.org
>
> Hi All,
> We are testing the BTS functionality. It seems as if the searches
> are ignoring
> strings with leading numbers.
>
> We have a name field indexed with BTS.
>
> A search for "where name LIKE '2STEP%'" returns 4 rows.
> The similar search "where bts_contains(name,'2STEP*')" returns ZERO rows.

>
> However a search for "where bts_contains(name,'STEP*')" returns multiple
rows
> beginning with STEP, **including the 4 rows beginning with 2STEP***!
>
> Is this how BTS searches are supposed to work? Is there a setting
> somewhere to
> tell BTS to include the leading numbers of a string?
>
> Thanks,
> Michael
>
>
>



>
>




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [337]

*******************************************************************************

---

Re: BTS not searching for numbers?

November 19, 2014, 10:13 am

≫ Next: Re: BTS not searching for numbers?

≪ Previous: Re: BTS not searching for numbers?

$

0

0

Hi Michael,

If you are using a version of BTS before bts.3.00, then yes, the Standard
Analyzer in Clucene on which BTS is based will behaves this way. With
bts.3.00, the analyzer feature was introduced to give the user more
control on how the text is analyzed and turned into tokens.

-- Mark.
Mark Ashworth
IBM Informix Extensibility Architect
Office phone: +1 (905) 413-5033
Alternate: +1 (905) 697-8094
Email: ashworth@ca.ibm.com
Check out my blog

From: "MICHAEL HOFFMAN"<mrh@panix.com>
To: datablade-list@iiug.org
Date: 11/19/2014 10:49 AM
Subject: BTS not searching for numbers? [336]
Sent by: datablade-list-bounces@iiug.org

Hi All,
We are testing the BTS functionality. It seems as if the searches are
ignoring
strings with leading numbers.

We have a name field indexed with BTS.

A search for "where name LIKE '2STEP%'" returns 4 rows.
The similar search "where bts_contains(name,'2STEP*')" returns ZERO rows.

However a search for "where bts_contains(name,'STEP*')" returns multiple
rows
beginning with STEP, **including the 4 rows beginning with 2STEP***!

Is this how BTS searches are supposed to work? Is there a setting
somewhere to
tell BTS to include the leading numbers of a string?

Thanks,
Michael









*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [338]

*******************************************************************************

Re: BTS not searching for numbers?

November 19, 2014, 10:36 am

≫ Next: Re: BTS not searching for numbers?

≪ Previous: Re: BTS not searching for numbers?

$

0

0

That's it! BTS 2.00 is installed. We're running Informix 11.5.FC9.

So when we jump to Informix 12.1, we'll be able to implement BTS 3.00, or
later?

Thanks,
Mike




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [339]

*******************************************************************************

Re: BTS not searching for numbers?

November 19, 2014, 10:44 am

≫ Next: REMOVE

≪ Previous: Re: BTS not searching for numbers?

$

0

0

BTS 2.00
Informix 11.5.FC9

So, running queries with digit-led strings breaks the C-Lucene engine, but it
seems ONLY when using a wildcard.

"where bts_contains(name,"2STEP*")" returns ZERO rows.
"where bts_contains(name,"2STEP")" returns all of the desired rows.

Why would a **trailing** wildcard affect the **leading** character of a
string?
Without the wildcard, C-Lucene/BTS has no problem finding just the rows that
start with the digit. But adding the wildcard, all of a sudden, the blade gets
lost.

That is quite strange!

Thanks,
Mike




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [340]

*******************************************************************************

REMOVE

November 19, 2014, 11:06 am

≫ Next: Very bad BTS insert performance

≪ Previous: Re: BTS not searching for numbers?

$

0

0

=0D=0A=0D=0A-----Original Message-----=0D=0AFrom: datablade-list-bounces@ii=
ug=2Eorg [mailto:datablade-list-bounces@iiug=2Eorg] On Behalf Of MICHAEL HO=
FFMAN=0D=0ASent: Wednesday, November 19, 2014 1:44 PM=0D=0ATo: datablade-li=
st@iiug=2Eorg=0D=0ASubject: Re: BTS not searching for numbers? [340]=0D=0A=
=0D=0ABTS 2=2E00=0D=0AInformix 11=2E5=2EFC9 =0D=0A=0D=0ASo, running queries=
with digit-led strings breaks the C-Lucene engine, but it seems ONLY when =
using a wildcard=2E =0D=0A=0D=0A"where bts_contains(name,"2STEP*")" returns=
ZERO rows=2E =0D=0A"where bts_contains(name,"2STEP")" returns all of the d=
esired rows=2E =0D=0A=0D=0AWhy would a **trailing** wildcard affect the **l=
eading** character of a string? =0D=0AWithout the wildcard, C-Lucene/BTS ha=
s no problem finding just the rows that start with the digit=2E But adding =
the wildcard, all of a sudden, the blade gets lost=2E =0D=0A=0D=0AThat is q=
uite strange! =0D=0A=0D=0AThanks,=0D=0AMike =0D=0A=0D=0A=0D=0A*************=
****************************************************************** =0D=0A =
2E =0D=
=0A=0D=0AThis e-mail may contain confidential or privileged information=2E =
If=0D=0Ayou think you have received this e-mail in error, please advise the=
=0D=0Asender by reply e-mail and then delete this e-mail immediately=2E=0D=
=0AThank you=2E Aetna




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [341]

*******************************************************************************

Very bad BTS insert performance

November 28, 2014, 4:53 am

≫ Next: Re: Very bad BTS insert performance

≪ Previous: REMOVE

$

0

0

Hello all,

I'm currently working on migrating an external Lucene index into an IFX DB
(11.70.FC8W1 on AIX) using BTS 3.00.
Because we cannot rewrite the whole client application logic, data has to be
split into 13 tables having a total of 17 BTS indexes on VARCHAR and LVARCHAR
columns.

While search performance is very good, adding data to this table structure is
unacceptable slow. It takes only about 2 seconds to load one day's data into
the external index, but anything between 50 and 130 seconds with BTS!

The amount of data for each day is roughly the same, so it's not
comprehensible why the performance differs so much even for the same day
loaded repeatedly.

If I disable all BTS indexes, load speed for one day is 10-15 seconds (still
too much but there are some optimization possibilites in the loader).

BTS indexes are all defined like this:
CREATE INDEX i_ecas_cn ON env_casts
(

cast_name bts_varchar_ops
)
USING BTS (canonical_maps="({ü}:{ue},{ä}:{ae},{ö}:{oe},{ß}:{ss})",
stopwords="(der,die,das,den,des,und,um,ist,im,am)", tempspace='sblob_dbs01',
delete="deferred",max_clause_count="2048", xact_ramdirectory ="yes")
IN bts_sbspace;

Currenty I have created 5 BTS VPs, but I noticed that regardless how much are
running only one of them is used when data is inserted (monitored with OAT and
ServerStudio).

I already upgraded from 11.50/BTS 2.00 but performance gain is negligible and
nowhere near the gains described in other threads here.
Is there anything to change in index definition or system configuration to
bring performance to the level of the external index, or is this (lack of)
speed just "normal" with my number of tables and indexes?

Thanks in advance,
Jens




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [342]

*******************************************************************************

Re: Very bad BTS insert performance

November 28, 2014, 6:37 am

≫ Next: Re: Very bad BTS insert performance

≪ Previous: Very bad BTS insert performance

$

0

0

IB that the BTS VPs will not be used in parallel unless PDQPRIORITY is set
> 0. Also, does your load job use individual insert statements? A
prepared statement with replaceable parameters executed for each row? To
seriously optimize insert performance you need to use a PUT cursor,
increase the size of the communication buffer that PUT uses (FET_BUF_SIZE)
so that it holds a thousand or more rows, and set PDQPRIORITY to
parallelize the BTS index maintenance.

Art

Art S. Kagel, President and Principal Consultant
ASK Database Management
www.askdbmgt.com

Blog: http://informix-myview.blogspot.com/

Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.

On Fri, Nov 28, 2014 at 7:53 AM, JENS SCHACHERL <jens.schacherl@de.ibm.com>
wrote:

> Hello all,
>
> I'm currently working on migrating an external Lucene index into an IFX DB
> (11.70.FC8W1 on AIX) using BTS 3.00.
> Because we cannot rewrite the whole client application logic, data has to
> be
> split into 13 tables having a total of 17 BTS indexes on VARCHAR and
> LVARCHAR
> columns.
>
> While search performance is very good, adding data to this table structure
> is
> unacceptable slow. It takes only about 2 seconds to load one day's data
> into
> the external index, but anything between 50 and 130 seconds with BTS!
>
> The amount of data for each day is roughly the same, so it's not
> comprehensible why the performance differs so much even for the same day
> loaded repeatedly.
>
> If I disable all BTS indexes, load speed for one day is 10-15 seconds
> (still
> too much but there are some optimization possibilites in the loader).
>
> BTS indexes are all defined like this:
> CREATE INDEX i_ecas_cn ON env_casts
> (
>
> cast_name bts_varchar_ops
> )
> USING BTS (canonical_maps="({ü}:{ue},{ä}:{ae},{ö}:{oe},{ß}:{ss})",
> stopwords="(der,die,das,den,des,und,um,ist,im,am)",
> tempspace='sblob_dbs01',
> delete="deferred",max_clause_count="2048", xact_ramdirectory ="yes")
> IN bts_sbspace;
>
> Currenty I have created 5 BTS VPs, but I noticed that regardless how much
> are
> running only one of them is used when data is inserted (monitored with OAT
> and
> ServerStudio).
>
> I already upgraded from 11.50/BTS 2.00 but performance gain is negligible
> and
> nowhere near the gains described in other threads here.
> Is there anything to change in index definition or system configuration to
> bring performance to the level of the external index, or is this (lack of)
> speed just "normal" with my number of tables and indexes?
>
> Thanks in advance,
> Jens
>
>
>
>

>
>
>

--001a11c33e907aae6e0508ec3190




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [343]

*******************************************************************************

Re: Very bad BTS insert performance

November 28, 2014, 8:12 am

≫ Next: Inserting irregular timestamps into Time Series

≪ Previous: Re: Very bad BTS insert performance

$

0

0

Hello Art,

thanks for your help, I try to increase PDQPRIO first.
The loader is a client-server-app running on Websphere 8 and AFAIK works as
you suspected, but I check that next week with it's developer and we'll set
FET_BUF_SIZE=32767 and IFX_USEPUT=1 for the JDBC connection, if missing.

Regards, Jens




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [344]

*******************************************************************************

---

Inserting irregular timestamps into Time Series

November 30, 2014, 4:43 pm

≫ Next: Re: Inserting irregular timestamps into Time S....

≪ Previous: Re: Very bad BTS insert performance

$

0

0

Hi

Just setting up a Timeseries for the first time for holding regular 30-second
data, and not sure how to handle timestamps which don't align to the calendar
boundaries.

I have a calendar pattern that looks like:

INSERT INTO CalendarPatterns values ( 'stan2_30sec', '{1 on , 29
off},second');

And a table which looks like:

create row type tagsample(
sample_dt datetime year to fraction(5),
sample_value float
);

create table tagsamples (
tag_id int,
sample TimeSeries(tagsample));

INSERT INTO tagsamples VALUES(1, "origin(2014-01-01 00:00:00.00000),
calendar(stan2_30sec), container(stan_tagsamples),
threshold(0),irregular,[]");

I have seven years worth of 30-second data in a relational table, but the data
doesn't line up with the calendar boundaries (eg, could be at 10:00:25,
10:00:55, 10:01:25 - rather than 10:00:30, 10:01:00, 10:01:30);

Is there a way to do an insert statement which automatically 'rounds' the time
to the nearest calendar boundary? (eg, would interpret an insert with
"2014-11-28 16:20:05" as "2014-11-28 16:20:00"

Thanks very much
Jay




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [345]

*******************************************************************************

Re: Inserting irregular timestamps into Time S....

November 30, 2014, 4:51 pm

≫ Next: Re: Inserting irregular timestamps into Time S....

≪ Previous: Inserting irregular timestamps into Time Series

$

0

0

What do you see when you insert it as is ?

Cheers
Paul

Paul Watson
Oninit www.oninit.com
+1 913 387 7529

> On Nov 30, 2014, at 18:43, JAY BAKER <jay.baker@fonterra.com> wrote:
>
> Hi
>
> Just setting up a Timeseries for the first time for holding regular
30-second
> data, and not sure how to handle timestamps which don't align to the
calendar
> boundaries.
>
> I have a calendar pattern that looks like:
>
> INSERT INTO CalendarPatterns values ( 'stan2_30sec', '{1 on , 29
> off},second');
>
> And a table which looks like:
>
> create row type tagsample(
> sample_dt datetime year to fraction(5),
> sample_value float
> );
>
> create table tagsamples (
> tag_id int,
> sample TimeSeries(tagsample));
>
> INSERT INTO tagsamples VALUES(1, "origin(2014-01-01 00:00:00.00000),
> calendar(stan2_30sec), container(stan_tagsamples),
> threshold(0),irregular,[]");
>
> I have seven years worth of 30-second data in a relational table, but the
data
> doesn't line up with the calendar boundaries (eg, could be at 10:00:25,
> 10:00:55, 10:01:25 - rather than 10:00:30, 10:01:00, 10:01:30);
>
> Is there a way to do an insert statement which automatically 'rounds' the
time
> to the nearest calendar boundary? (eg, would interpret an insert with
> "2014-11-28 16:20:05" as "2014-11-28 16:20:00"
>
> Thanks very much
> Jay
>
>
>

>




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [346]

*******************************************************************************

Re: Inserting irregular timestamps into Time S....

December 1, 2014, 6:52 am

≫ Next: Re: Very bad BTS insert performance

≪ Previous: Re: Inserting irregular timestamps into Time S....

$

0

0

Hi Jay,

It sounds like the data is more regular than irregular?

When you create a calendar with off periods, elements on the off period
cannot be inserted. If you need a strict calendar that will only allow
elements on the 5 and 35 minutes, try using
the pattern:

pattern({5 off, 1 on, 24 off} minute)

for data on the 5 and 35 minute period

If you do not need such a strict calendar, you could use the pattern:
pattern({1 on} minute)
and should work for both the data coming in on at 00:00 and 30:00 or 05:00
and 35:00. However this calendar would not prevent elements from being
entered that are not on that boundary.

-- Mark.
Mark Ashworth
IBM Informix Extensibility Architect
Office phone: +1 (905) 413-5033
Alternate: +1 (905) 697-8094
Email: ashworth@ca.ibm.com
Check out my blog

From: "JAY BAKER"<jay.baker@fonterra.com>
To: datablade-list@iiug.org
Date: 11/30/2014 07:43 PM
Subject: Inserting irregular timestamps into Time Series [345]
Sent by: datablade-list-bounces@iiug.org

Hi

Just setting up a Timeseries for the first time for holding regular
30-second
data, and not sure how to handle timestamps which don't align to the
calendar
boundaries.

I have a calendar pattern that looks like:

INSERT INTO CalendarPatterns values ( 'stan2_30sec', '{1 on , 29
off},second');

And a table which looks like:

create row type tagsample(
sample_dt datetime year to fraction(5),
sample_value float
);

create table tagsamples (
tag_id int,
sample TimeSeries(tagsample));

INSERT INTO tagsamples VALUES(1, "origin(2014-01-01 00:00:00.00000),
calendar(stan2_30sec), container(stan_tagsamples),
threshold(0),irregular,[]");

I have seven years worth of 30-second data in a relational table, but the
data
doesn't line up with the calendar boundaries (eg, could be at 10:00:25,
10:00:55, 10:01:25 - rather than 10:00:30, 10:01:00, 10:01:30);

Is there a way to do an insert statement which automatically 'rounds' the
time
to the nearest calendar boundary? (eg, would interpret an insert with
"2014-11-28 16:20:05" as "2014-11-28 16:20:00"

Thanks very much
Jay









*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [347]

*******************************************************************************

Re: Very bad BTS insert performance

December 3, 2014, 6:20 am

≫ Next: my new article

≪ Previous: Re: Inserting irregular timestamps into Time S....

$

0

0

Hello,

we have set FET_BUF_SIZE=32767, IFX_PUT=1 and PDQPRIORITY=HIGH for the data
source used by the loader client.
Overall load time is now 1.5-2 times better than before but that is still way
apart from performance of the external Lucene index.

Are there any more optimization options, or is the lack of speed simply a
consequence of our number of indexes and tables?
Overall amount of data per day is not that much, around 2000-3000 rows to
INSERT with an equal number of DELETEs.

Regards,
Jens




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [348]

*******************************************************************************

my new article

September 21, 2016, 10:07 am

≫ Next: How can I create BTS index?

≪ Previous: Re: Very bad BTS insert performance

$

0

0

Hi,

Have you already read my new article about the elections? You've got to take a
look, here is the link <http://balance.myhealthtool.com/e4bgdip>

k7nirmala




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [349]

*******************************************************************************

How can I create BTS index?

December 2, 2016, 5:32 am

≫ Next: Re: How can I create BTS index?

≪ Previous: my new article

$

0

0

Hello,

I have problem with creating BTS (basic text search) index in Informix.
I have created test table and sbspace to store index information just
like it is in documentation. Now my sbspace looks like:

[informix@test-informix informix]$ onspaces -c -S bts_sbspace -p
/home/informix/chunks/bts_sbspace -o 0 -s 1500000 -Df "LOGGING=ON"
....
[informix@test-informix informix]$ onstat -d | grep SB
ed682820 7 0x68001 7 3 2048 N SB
informix sbspace
fd409e00 14 0x68001 73 1 2048 N SB
informix bts_sbspace
ed8c1028 7 7 0 50000 43896 44061
POSBD /home/informix/IFMXDATA/ol_testifx/sbspacedbs.000
edb82db8 30 7 0 500000 466235 466319
POSBD /home/informix/IFMXDATA/ol_testifx/sbspacedbs.001
ee7fe028 31 7 0 500000 466306 466319
POSBD /home/informix/IFMXDATA/ol_testifx/sbspacedbs.002
fd40de00 73 14 0 750000 699446 699446
POSBD /home/informix/chunks/bts_sbspace

But when I want to create index:

CREATE INDEX lab1_bts_index ON lab1(text_data bts_lvarchar_ops) USING
BTS IN bts_sbspace;

it ends with error:

Error: bts internal error. File c/bts_am_extspace.c, line 505
SQLState: BTS02
ErrorCode: -937
Position: 88

I use `IBM Informix Dynamic Server Version 11.50.FC4` with `bts2.0`
registered for test database.

I think something is wrong with bts_sbspace, but my version 11.50.FC4 do
not show it correctly. Maybe it is related to the problem fixed in
11.50.FC6:

http://www-01.ibm.com/support/docview.wss?uid=swg27017619

IC63852 BTS DOES NOT LOG USEFUL ERRORS FOR CREATE BTS INDEX FAILURE WHEN
SBSPACETEMP IS CONFIGURED INCORRECTLY

How can I create BTS index?

I asked this also on StackOverflow:

http://stackoverflow.com/questions/40912439/how-to-create-bts-index-in-informix

--
Regards,
Michał




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [350]

*******************************************************************************

Re: How can I create BTS index?

December 2, 2016, 12:52 pm

≫ Next: Re: How can I create BTS index?

≪ Previous: How can I create BTS index?

$

0

0

Hi Michal,

Error: bts internal error. File c/bts=5Fam=5Fextspace.c, line 505=20

is generated when it cannot create structures in the default sbspace.=20

The IN storage clause on the CREATE INDEX statement specifies where the=20
index is stored
but there are also structures built in the default sbspace specified by=20
the SBSPACENAME=20
onconfig variable.

These errors usually happen if either the SBSPACENAME variable has changed
since the BTS blade was registered or it was not created as a logged=20
sbspace.

Check your configuration that the SBSPACENAME onconfig variable is set and
points at a logged sbspace. Also use oncheck to verify there are no=20
problems with
the default sbspace.

-- Mark.
Mark Ashworth=20
Office phone: +1 (905) 413-5033=20
Email: mailto:ashworth@ca.ibm.com=20
Check out my blog=20

From: "MICHAL NIKLAS"<michal.niklas@heuthes.pl>
To: datablade-list@iiug.org
Date: 2016/12/02 08:33 AM
Subject: How can I create BTS index? [350]
Sent by: datablade-list-bounces@iiug.org

Hello,=20

I have problem with creating BTS (basic text search) index in Informix.=20
I have created test table and sbspace to store index information just=20
like it is in documentation. Now my sbspace looks like:=20

[informix@test-informix informix]$ onspaces -c -S bts=5Fsbspace -p=20
/home/informix/chunks/bts=5Fsbspace -o 0 -s 1500000 -Df "LOGGING=3DON"=20
.....=20
[informix@test-informix informix]$ onstat -d | grep SB=20
ed682820 7 0x68001 7 3 2048 N SB=20
informix sbspace=20
fd409e00 14 0x68001 73 1 2048 N SB=20
informix bts=5Fsbspace=20
ed8c1028 7 7 0 50000 43896 44061=20
POSBD /home/informix/IFMXDATA/ol=5Ftestifx/sbspacedbs.000=20
edb82db8 30 7 0 500000 466235 466319=20
POSBD /home/informix/IFMXDATA/ol=5Ftestifx/sbspacedbs.001=20
ee7fe028 31 7 0 500000 466306 466319=20
POSBD /home/informix/IFMXDATA/ol=5Ftestifx/sbspacedbs.002=20
fd40de00 73 14 0 750000 699446 699446=20
POSBD /home/informix/chunks/bts=5Fsbspace=20

But when I want to create index:=20

CREATE INDEX lab1=5Fbts=5Findex ON lab1(text=5Fdata bts=5Flvarchar=5Fops) U=
SING=20
BTS IN bts=5Fsbspace;=20

it ends with error:=20

Error: bts internal error. File c/bts=5Fam=5Fextspace.c, line 505=20
SQLState: BTS02=20
ErrorCode: -937=20
Position: 88=20

I use `IBM Informix Dynamic Server Version 11.50.FC4` with `bts2.0`=20
registered for test database.=20

I think something is wrong with bts=5Fsbspace, but my version 11.50.FC4 do =

not show it correctly. Maybe it is related to the problem fixed in=20
11.50.FC6:=20

http://www-01.ibm.com/support/docview.wss?uid=3Dswg27017619=20

IC63852 BTS DOES NOT LOG USEFUL ERRORS FOR CREATE BTS INDEX FAILURE WHEN=20
SBSPACETEMP IS CONFIGURED INCORRECTLY=20

How can I create BTS index?=20

I asked this also on StackOverflow:=20

http://stackoverflow.com/questions/40912439/how-to-create-bts-index-in-info=
rmix=20

--=20
Regards,=20
Michał=20

***************************************************************************=
****=20

=20




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [351]

*******************************************************************************

---

Re: How can I create BTS index?

December 5, 2016, 1:31 am

≫ Next: Re: How can I create BTS index?

≪ Previous: Re: How can I create BTS index?

$

0

0

Info about my configuration in ONCONFIG:

[informix@test-informix extend]$ grep SBSPA
/opt/IBM/informix/etc/onconfig.ol_testifx

SBSPACETEMP temp_sbspace

SBSPACENAME sbspace

SYSSBSPACENAME

CDR_QDATA_SBSPACE

I created bts_sbspace and temp_sbspace using:

onspaces -c -S bts_sbspace -p /home/informix/chunks/bts_sbspace -o 0 -s
1500000 -Df "LOGGING=ON"

onspaces -c -S temp_sbspace -t -o 0 -s 50000 -p
/home/informix/chunks/temp_sbspace

Info about sbspaces:

[informix@test-informix extend]$ oncheck -cS

Validating space 'sbspace' ...

sbspace Metadata Partition Partnum Used Free

sbspace:'informix'.TBLSpace 0x700001 10 40

sbspace:'informix'.sbspace_desc 0x700002 2 2

sbspace:'informix'.chunk_adjunc 0x700003 4 0

sbspace:'informix'.LO_ud_free 0x700004 2 127

sbspace:'informix'.LO_hdr_partn 0x700005 11472 0

sbspace:'informix'.LO_ud_free 0x700006 2 1294

sbspace:'informix'.LO_hdr_partn 0x700007 14172 0

sbspace:'informix'.LO_ud_free 0x700008 2 1294

sbspace:'informix'.LO_hdr_partn 0x700009 11000 3172

Large Objects

ID Ref Size Allocced Creat Last

Sbs# Chk# Seq# Cnt (Bytes) Pages Extns Flags Modified

---- ---- ----- ---- ---------- -------- ----- ----- ------------------------

7 7 1 1 3411 2 1 L-N-H Sun Dec 11 22:49:59 2011

....

7 7 2999 1 3030 2 1 L-N-H Wed Jul 22 10:06:05 2015

7 7 29565 1 17 1 1 L-N-H Wed Jul 22 10:04:13 2015

Large Objects

ID Ref Size Allocced Creat Last

Sbs# Chk# Seq# Cnt (Bytes) Pages Extns Flags Modified

---- ---- ----- ---- ---------- -------- ----- ----- ------------------------

7 30 1 1 3030 2 1 L-N-H Wed Jul 22 10:06:03 2015

....

7 30 44481 1 3030 2 1 L-N-H Wed Jul 22 10:06:05 2015

7 30 44482 1 51936 26 1 L-N-H Wed Jul 22 10:04:12 2015

Large Objects

ID Ref Size Allocced Creat Last

Sbs# Chk# Seq# Cnt (Bytes) Pages Extns Flags Modified

---- ---- ----- ---- ---------- -------- ----- ----- ------------------------

7 31 1 1 100 1 1 L-N-H Wed Jul 9 12:52:47 2014

....

7 31 7 1 5963 3 1 L-N-H Sat Nov 7 13:52:20 2015

7 31 8 1 3411 2 1 L-N-H Sat Nov 7 13:52:20 2015

Validating space 'bts_sbspace' ...

sbspace Metadata Partition Partnum Used Free

bts_sbspace:'informix'.TBLSpace 0xe00001 6 44

bts_sbspace:'informix'.sbspace_desc 0xe00002 2 2

bts_sbspace:'informix'.chunk_adjunc 0xe00003 2 2

bts_sbspace:'informix'.LO_ud_free 0xe00004 2 1941

bts_sbspace:'informix'.LO_hdr_partn 0xe00005 2 10969

Validating space 'temp_sbspace' ...

sbspace Metadata Partition Partnum Used Free

temp_sbspace:'informix'.TBLSpace 0xf00001 2 48

temp_sbspace:'informix'.sbspace_desc 0xf00002 1 3

temp_sbspace:'informix'.chunk_adjunc 0xf00003 1 3

temp_sbspace:'informix'.LO_ud_free 0xf00004 1 64

temp_sbspace:'informix'.LO_hdr_partn 0xf00005 1 363

I wanted to unregister BTS, but without success:

[informix@test-informix extend]$ ls

binaryudt.1.0 bts.2.00 ifxbuiltins.1.1 ifxmngr ifxrltree.2.00 jars krakatoa
LLD.1.20.FC2 mc mqblade.2.0 Node.2.0 wfs.1.00.FC1

[informix@test-informix extend]$ blademgr

ol_testifx>unregister bts.2.00 multi

Unregister module bts.2.00 out of database multi? [Y/n]Y

blademgr: ERROR: DataBlade bts.2.00 could not be unregistered from database
multi.

Use 'show last log' for details of failures in module's scripts.

ol_testifx>ol_testifx>show_last_log

Usage is: show databases|client|logs|modules|servers.

ol_testifx>show last log

Mack Informix-Manager DataBlade Module Version 5.00 - Apr 14 2009

blade: ifxmngr

err expct: expected

SQL stmt: insert into informix.systraceclasses(name)| values ('blademgr');

SQL state: X23000:-239 : Could not insert new row - duplicate value in a
UNIQUE INDEX column (Unique Index:informix.systraceclasses).XIX000:-100 : :
ISAM error: duplicate value for a record with unique key.

.....

blade: ifxmngr

err expct: expected

SQL stmt: insert into sysbldregistered select * from bld_registered;

Type 'Enter' for next page, or 'Q' and 'Enter' to quit. :-

SQL state: X23000:-268 : Unique constraint (informix.u112_12)
violated.XIX000:-100 : : ISAM error: duplicate value for a record with unique
key.

blade: ifxmngr

err expct: expected

SQL stmt: drop table bld_registered;

SQL state: XIX000:-523 : Can only recover, repair, truncate or drop table.

blade: ifxmngr

err expct: expected

SQL stmt: create view bld_registered as select * from sysbldregistered;

SQL state: XS0001:-310 : Table (informix.bld_registered) already exists in
database.

blade: ifxmngr

err expct: expected

SQL stmt: insert into sysbldirequired select * from bldi_required;

SQL state: X42000:-206 : The specified table (bldi_required) is not in the
database.XIX000:-111 : : ISAM error: no record found.

blade: ifxmngr

err expct: expected

SQL stmt: drop table bldi_required;

Type 'Enter' for next page, or 'Q' and 'Enter' to quit. :-

SQL state: X42000:-206 : The specified table (bldi_required) is not in the
database.XIX000:-111 : : ISAM error: no record found.

blade: ifxmngr

err expct: expected

SQL stmt: drop table bldi_provided;

SQL state: XIX000:-523 : Can only recover, repair, truncate or drop table.

blade: ifxmngr

err expct: expected

SQL stmt: create view bldi_provided as select * from sysbldiprovided;

SQL state: XS0001:-310 : Table (informix.bldi_provided) already exists in
database.

But I can register BTS (replace it):

ol_testifx>register bts.2.00 multi

Register module bts.2.00 into database multi? [Y/n]Y

DataBlade bts.2.00 is already registered in database multi.

DataBlade module(s) that will be replaced by bts.2.00:

bts.2.00

Are these changes ok? [Y/n]Registering DataBlade module... (may take a while).

DataBlade bts.2.00 was successfully registered in database multi.

ol_testifx>

Error while creating index is the same:

CREATE INDEX lab1_bts_index ON lab1(text_data bts_lvarchar_ops) USING BTS IN
bts_sbspace

bts internal error. File c/bts_am_extspace.c, line 505

--
Regards,
Michał




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [352]

*******************************************************************************

Re: How can I create BTS index?

December 5, 2016, 9:51 pm

≫ Next: RE: Size limits in basic text search?

≪ Previous: Re: How can I create BTS index?

$

0

0

Hello,

More info about my configuration:

[informix@test-informix ~]$ onstat -g sch

IBM Informix Dynamic Server Version 11.50.FC4 -- On-Line -- Up 20:25:21 --
3260628 Kbytes

VP Scheduler Statistics:
vp pid class semops busy waits spins/wait
1 28550 cpu 4833639 4879176 9951
2 28551 adm 0 0 0
3 28552 bts 15 15 10001
4 28553 cpu 4743167 4782904 9957
5 28554 cpu 1934334 1958098 9937
6 28555 lio 2 0 0
7 28556 pio 2 0 0
8 28557 aio 4553 0 0
9 28558 msc 8089 0 0
10 28559 aio 17 0 0
11 28560 soc 8 8 1000
12 28561 soc 5 5 1000
13 28562 soc 10 10 1000
14 28563 soc 4 4 1000
15 28564 aio 7 0 0
16 28565 aio 8 0 0
17 28566 aio 4 0 0

Thread Migration Statistics:
vp pid class steal-at steal-sc idlvp-at idlvp-sc inl-polls Q-ln
1 28550 cpu 5083704 947044 723731 621993 0 0
2 28551 adm 0 0 698453 84947 0 0
3 28552 bts 0 0 2 2 0 0
4 28553 cpu 4928365 803627 631818 540807 0 0
5 28554 cpu 2138105 544372 453150 377971 0 0
6 28555 lio 0 0 0 0 0 0
7 28556 pio 0 0 0 0 0 0
8 28557 aio 0 0 5 5 0 0
9 28558 msc 0 0 860 464 0 0
10 28559 aio 0 0 1 1 0 0
11 28560 soc 0 0 84907 65063 0 0
12 28561 soc 0 0 89267 68498 0 0
13 28562 soc 0 0 83437 63751 0 0
14 28563 soc 0 0 85364 64226 0 0
15 28564 aio 0 0 1 1 0 0
16 28565 aio 0 0 0 0 0 0
17 28566 aio 0 0 0 0 0 0

[informix@test-informix ~]$ onstat -c |grep VPCLASS
VPCLASS cpu,num=3,noage
JVPCLASSPATH
$INFORMIXDIR/extend/krakatoa/krakatoa.jar:$INFORMIXDIR/extend/krakatoa/jdbc.jar
VPCLASS bts,noyield,num=1

[informix@test-informix ~]$ onstat -version
Program Name: onstat
Build Version: 11.50.FC4
Build Number: N186
Build Host: vidar
Build OS: Linux 2.6.9-34.ELsmp
Build Date: Tue Apr 14 22:03:57 CDT 2009
GLS Version: glslib-4.50.FC5

[informix@test-informix ~]$ echo "select * from btsfse_storage;" |dbaccess -e
multi -
Database selected.

select * from btsfse_storage;
206: The specified table (btsfse_storage) is not in the database.

111: ISAM error: no record found.
Error in line 1
Near character position 28

Database closed.

--
Regards,
Michał




*******************************************************************************

To post a response via email (IIUG members only):

1. Address it to datablade-list@iiug.org
2. Include the bracketed message number in the subject line: [353]

*******************************************************************************

RE: Size limits in basic text search?

February 22, 2013, 3:00 pm

≫ Next: Re: RE: Size limits in basic text search?

≪ Previous: Re: How can I create BTS index?

$

0

0

Hi Ingvi Stigsson, BTS is based on the CLucene Text Search engine. CLucene has a default maximum number of words (tokens) that it will index in on field of document. Without XML, there is only one field: contents. With xml each tag indexed is a field. This default value per field is 10000. If you have an average of 5 character words, this would mean about 50,000 characters would be indexed (the count does not include with whitespace, punctuation or stopwords). There is supposed to be a warning generated if that limit is exceeded. The warning reads as follows: Indexing a huge number of tokens form a single field (<field name>, in this case) can cause CLucene to use memory excessively. By default CLucene will accept only 10000 tokens... I will need to look more into why it may not be output to the user. What is your machine platform and IDS version? -- BTS has a parameter, field_token_max (which is currently undocumented), that can change the default (up or down) for this parameter: field_token_max="n" where n is the maximum number of tokens to index in a given field. The physical maximum is a signed 32bit integer: 2^31-1. For example: create index my_bts_idx on bts_tab(text bts_char_ops) using bts(field_token_max="40000") in bts_sbspace; Please note that increasing this value may cause BTS to use significantly more shared memory. Mark Ashworth IBM Informix Extensibility Architect -----Original Message----- From: datablade-list-bounces@iiug.org [mailto:datablade-list-bounces@iiug.org] On Behalf Of INGVI STIGSSON Sent: February 20, 2013 12:51 PM To: datablade-list@iiug.org Subject: Size limits in basic text search? [277] Hi I have been trying the Basic Text Search datablade. Basically I am converting html and pdf to text files which I import into clob which is indexed with BTS. However in my testing the whole text is not indexed, only first part of it. In one test I had a text file of about 1,2M, but only the first 53K seems to be indexed. Is this to be expected? Is it possible to change this in configuration? Best regards Ingvi Stigsson **************************************************************************** *** ******************************************************************************* To post a response via email (IIUG members only): 1. Address it to datablade-list@iiug.org 2. Include the bracketed message number in the subject line: [283] *******************************************************************************

Re: RE: Size limits in basic text search?

February 25, 2013, 5:40 am

≫ Next: Re: Size limits in basic text search?

≪ Previous: RE: Size limits in basic text search?

$

0

0

Hi Mark Thanks for your explanation. I will play with this parameter and see how far I can go with it in my testing. I am using IBM Informix Dynamic Server Version 11.70.FC5IE (Innovator C) bts.3.00 OS: Centos Linux 2.6.32-220.4.2.el6.x86_64 Best regards Ingvi Stigsson ******************************************************************************* To post a response via email (IIUG members only): 1. Address it to datablade-list@iiug.org 2. Include the bracketed message number in the subject line: [284] *******************************************************************************

Re: Size limits in basic text search?

February 25, 2013, 5:43 am

≫ Next: BTS47 error

≪ Previous: Re: RE: Size limits in basic text search?

$

0

0

Hi Claus and thanks for your responses. I will check the settings for huge/large pages. But Mark Ashworth gave me some hints on using field_token_max parameter when creating the index. Best regards Ingvi Stigsson ******************************************************************************* To post a response via email (IIUG members only): 1. Address it to datablade-list@iiug.org 2. Include the bracketed message number in the subject line: [285] *******************************************************************************