Ivory: Experiments

These regression runs represent experiments presented in Wang et al.'s CIKM 2010 paper Ranking under Temporal Constraints.

Wt10g results

# command-line
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/wt10g/run.wt10g.CIKM2010.title.indep.xml data/wt10g/wt10g_queries_501-550.xml
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/wt10g/run.wt10g.CIKM2010.title.joint.xml data/wt10g/wt10g_queries_501-550.xml
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/wt10g/run.wt10g.CIKM2010.desc.indep.xml data/wt10g/wt10g_queries_501-550_desc.xml
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/wt10g/run.wt10g.CIKM2010.desc.joint.xml data/wt10g/wt10g_queries_501-550_desc.xml
 
# evaluating effectiveness
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-indep-x1.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-indep-x1.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-indep-x2.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-indep-x2.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-indep-x3.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-indep-x3.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-indep-x4.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-indep-x4.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-indep-x5.0.txt

etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-joint-x1.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-joint-x1.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-joint-x2.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-joint-x2.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-joint-x3.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-joint-x3.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-joint-x4.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-joint-x4.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-title-joint-x5.0.txt

etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-indep-x1.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-indep-x1.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-indep-x2.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-indep-x2.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-indep-x3.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-indep-x3.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-indep-x4.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-indep-x4.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-indep-x5.0.txt

etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-joint-x1.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-joint-x1.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-joint-x2.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-joint-x2.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-joint-x3.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-joint-x3.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-joint-x4.0.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-joint-x4.5.txt
etc/trec_eval data/wt10g/qrels.wt10g ranking.cikm2010-wt10g-desc-joint-x5.0.txt

# junit
etc/junit.sh ivory.regression.cikm2010.Wt10g_Title_Indep
etc/junit.sh ivory.regression.cikm2010.Wt10g_Title_Joint
etc/junit.sh ivory.regression.cikm2010.Wt10g_Desc_Indep
etc/junit.sh ivory.regression.cikm2010.Wt10g_Desc_Joint

	Title		Description
Condition	Indep	Joint	Indep	Joint
QL x 1.0	0.1695	0.2200	0.1753	0.1778
QL x 1.5	0.2143	0.2230	0.1859	0.1937
QL x 2.0	0.2292	0.2325	0.1957	0.2048
QL x 2.5	0.2299	0.2275	0.2048	0.2136
QL x 3.0	0.2333	0.2307	0.2094	0.2126
QL x 3.5	0.2323	0.2358	0.2101	0.2133
QL x 4.0	0.2366	0.2366	0.2155	0.2201
QL x 4.5	0.2413	0.2378	0.2178	0.2226
QL x 5.0	0.2422	0.2387	0.2173	0.2175

Gov2 results

# command-line
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/gov2/run.gov2.CIKM2010.title.indep.xml data/gov2/queries.gov2.title.776-850.xml
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/gov2/run.gov2.CIKM2010.title.joint.xml data/gov2/queries.gov2.title.776-850.xml
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/gov2/run.gov2.CIKM2010.desc.indep.xml data/gov2/queries.gov2.desc.776-850.xml
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/gov2/run.gov2.CIKM2010.desc.joint.xml data/gov2/queries.gov2.desc.776-850.xml

# evaluating effectiveness
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-indep-x1.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-indep-x1.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-indep-x2.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-indep-x2.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-indep-x3.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-indep-x3.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-indep-x4.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-indep-x4.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-indep-x5.0.txt

etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-joint-x1.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-joint-x1.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-joint-x2.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-joint-x2.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-joint-x3.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-joint-x3.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-joint-x4.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-joint-x4.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-title-joint-x5.0.txt

etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-indep-x1.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-indep-x1.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-indep-x2.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-indep-x2.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-indep-x3.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-indep-x3.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-indep-x4.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-indep-x4.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-indep-x5.0.txt

etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-joint-x1.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-joint-x1.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-joint-x2.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-joint-x2.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-joint-x3.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-joint-x3.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-joint-x4.0.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-joint-x4.5.txt
etc/trec_eval data/gov2/qrels.gov2.all ranking.cikm2010-gov2-desc-joint-x5.0.txt

# junit
etc/junit.sh ivory.regression.cikm2010.Gov2_Title_Indep
etc/junit.sh ivory.regression.cikm2010.Gov2_Title_Joint
etc/junit.sh ivory.regression.cikm2010.Gov2_Desc_Indep
etc/junit.sh ivory.regression.cikm2010.Gov2_Desc_Joint

	Title		Description
Condition	Indep	Joint	Indep	Joint
QL x 1.0	0.1761	0.3174	0.2505	0.3067
QL x 1.5	0.2724	0.3201	0.2889	0.3089
QL x 2.0	0.3296	0.3361	0.3028	0.3115
QL x 2.5	0.3325	0.3364	0.3104	0.3104
QL x 3.0	0.3379	0.3506	0.3161	0.3277
QL x 3.5	0.3421	0.3522	0.3211	0.3260
QL x 4.0	0.3512	0.3524	0.3238	0.3335
QL x 4.5	0.3540	0.3524	0.3263	0.3302
QL x 5.0	0.3550	0.3524	0.3285	0.3294

Clue results

# command-line
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/clue/run.clue.CIKM2010.title.indep.xml data/clue/queries.web09.26-50.xml
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/clue/run.clue.CIKM2010.title.joint.xml data/clue/queries.web09.26-50.xml
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/clue/run.clue.CIKM2010.desc.indep.xml data/clue/queries.web09.26-50.desc.xml
etc/run.sh ivory.smrf.retrieval.RunQueryLocal data/clue/run.clue.CIKM2010.desc.joint.xml data/clue/queries.web09.26-50.desc.xml

# evaluating effectiveness
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-indep-x1.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-indep-x1.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-indep-x2.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-indep-x2.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-indep-x3.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-indep-x3.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-indep-x4.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-indep-x4.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-indep-x5.0.txt

etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-joint-x1.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-joint-x1.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-joint-x2.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-joint-x2.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-joint-x3.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-joint-x3.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-joint-x4.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-joint-x4.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-title-joint-x5.0.txt

etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-indep-x1.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-indep-x1.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-indep-x2.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-indep-x2.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-indep-x3.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-indep-x3.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-indep-x4.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-indep-x4.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-indep-x5.0.txt

etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-joint-x1.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-joint-x1.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-joint-x2.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-joint-x2.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-joint-x3.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-joint-x3.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-joint-x4.0.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-joint-x4.5.txt
etc/trec_eval data/clue/qrels.web09catB.txt ranking.cikm2010-clue-desc-joint-x5.0.txt

# junit
etc/junit.sh ivory.regression.cikm2010.Web09catB_Title_Indep
etc/junit.sh ivory.regression.cikm2010.Web09catB_Title_Joint
etc/junit.sh ivory.regression.cikm2010.Web09catB_Desc_Indep
etc/junit.sh ivory.regression.cikm2010.Web09catB_Desc_Joint

	Title		Description
Condition	Indep	Joint	Indep	Joint
QL x 1.0	0.1134	0.2256	0.1393	0.1683
QL x 1.5	0.1916	0.2256	0.1625	0.1563
QL x 2.0	0.2097	0.2306	0.1446	0.1573
QL x 2.5	0.2118	0.2315	0.1434	0.1558
QL x 3.0	0.2159	0.2324	0.1461	0.1561
QL x 3.5	0.2152	0.2348	0.1502	0.1567
QL x 4.0	0.2218	0.2360	0.1499	0.1587
QL x 4.5	0.2248	0.2360	0.1492	0.1567
QL x 5.0	0.2250	0.2360	0.1496	0.1561

(Note that results reported in the CIKM 2010 paper are StatMAP values, not standard MAP values reported in the table above.)

Experiments: RTC Model from CIKM 2010

Wt10g results

Gov2 results

Clue results