You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Rajat Gangwar <ra...@gmail.com> on 2017/04/02 18:43:20 UTC
re-emitting phantom batches
my topo is something like :
.partitionAggregate(..)
.groupBy(..)
.persistentAggregate(..)
Inserted 1 tuple, which got emitted with txn_id (batch_id) as 1.
But nothing happens after that. It just stuck and not even reaching
persistentAggregate.
This is what i see in logs
[image: Inline image 1]
I don't know what are these batch_ids which it is trying to re-emit.
any idea what could be the issue ?
Thanks,
Rajat
Re: re-emitting phantom batches
Posted by Rajat Gangwar <ra...@gmail.com>.
increased *topology.sleep.spout.wait.strategy.time.ms
<http://topology.sleep.spout.wait.strategy.time.ms>* value from 1 ms to 100
ms.
empty tuples are still there but now they are not bombarded and tuples are
getting processed as they come.
Is this the right way to handle dummy transactions ?
On Mon, Apr 3, 2017 at 3:44 PM, Rajat Gangwar <ra...@gmail.com>
wrote:
> This has slowed down the processing considerably.
> I inserted 1 tuple which got emitted with txn-id : *2141*.
>
> But when I check zookeeper node, currenttx is *1688*
> *[zk: 10.X.X.X:2181(CONNECTED) 76] get
> /storm_test/transactional/accrual_spout20/coordinator/currtx*
> *1688*
>
> and attempts in the queue :
> [*zk: 10.X.X.X:2181(CONNECTED) 77] get
> /storm_test/transactional/accrual_spout20/coordinator/currattempts*
>
> *{"1689":2,"1690":1,"1691":1,"1692":1,"1693":1,"1694":1,"1695":1,"1696":1,"1697":1,"1698":1,"1699":1,"1700":1,"1701":1,"1702":1,"1703":1,"1704":1,"1705":1,"1706":1,"1707":1,"1708":1,"1709":1,"1710":1,"1711":1,"1712":1,"1713":1,"1714":1,"1715":1,"1716":1,"1717":1,"1718":1,"1719":1,"1720":1,"1721":1,"1722":1,"1723":1,"1724":1,"1725":1,"1726":1,"1727":1,"1728":1,"1729":1,"1730":1,"1731":1,"1732":1,"1733":1,"1734":1,"1735":1,"1736":1,"1737":1,"1738":1,"1739":1,"1740":1,"1741":1,"1742":1,"1743":1,"1744":1,"1745":1,"1746":1,"1747":1,"1748":1,"1749":1,"1750":1,"1751":1,"1752":1,"1753":1,"1754":1,"1755":1,"1756":1,"1757":1,"1758":1,"1759":1,"1760":1,"1761":1,"1762":1,"1763":1,"1764":1,"1765":1,"1766":1,"1767":1,"1768":1,"1769":1,"1770":1,"1771":1,"1772":1,"1773":1,"1774":1,"1775":1,"1776":1,"1777":1,"1778":1,"1779":1,"1780":1,"1781":1,"1782":1,"1783":1,"1784":1,"1785":1,"1786":1,"1787":1,"1788":1,"1789":1,"1790":1,"1791":1,"1792":1,"1793":1,"1794":1,"1795":1,"1796":1,"1797":1,"1798":1,"1799":1,"1800":1,"1801":1,"1802":1,"1803":1,"1804":1,"1805":1,"1806":1,"1807":1,"1808":1,"1809":1,"1810":1,"1811":1,"1812":1,"1813":1,"1814":1,"1815":1,"1816":1,"1817":1,"1818":1,"1819":1,"1820":1,"1821":1,"1822":0,"1823":0,"1824":0,"1825":0,"1826":0,"1827":0,"1828":0,"1829":0,"1830":0,"1831":0,"1832":0,"1833":0,"1834":0,"1835":0,"1836":0,"1837":1,"1838":0,"1839":0,"1840":0,"1841":1,"1842":0,"1843":0,"1844":0,"1845":0,"1846":0,"1847":0,"1848":0,"1849":0,"1850":0,"1851":1,"1852":0,"1853":0,"1854":0,"1855":0,"1856":0,"1857":1,"1858":0,"1859":0,"1860":0,"1861":0,"1862":0,"1863":0,"1864":0,"1865":0,"1866":0,"1867":0,"1868":0,"1869":0,"1870":0,"1871":0,"1872":0,"1873":0,"1874":0,"1875":0,"1876":0,"1877":0,"1878":0,"1879":1,"1880":1,"1881":0,"1882":1,"1883":0,"1884":0,"1885":0,"1886":0,"1887":0,"1888":1,"1889":0,"1890":0,"1891":0,"1892":1,"1893":1,"1894":0,"1895":0,"1896":0,"1897":0,"1898":0,"1899":0,"1900":1,"1901":0,"1902":0,"1903":0,"1904":0,"1905":0,"1906":0,"1907":0,"1908":0,"1909":0,"1910":0,"1911":0,"1912":0,"1913":0,"1914":0,"1915":0,"1916":0,"1917":0,"1918":0,"1919":1,"1920":0,"1921":1,"1922":0,"1923":0,"1924":0,"1925":0,"1926":0,"1927":0,"1928":0,"1929":0,"1930":0,"1931":0,"1932":0,"1933":0,"1934":0,"1935":0,"1936":0,"1937":0,"1938":0,"1939":0,"1940":0,"1941":0,"1942":0,"1943":0,"1944":0,"1945":0,"1946":0,"1947":0,"1948":0,"1949":0,"1950":0,"1951":0,"1952":0,"1953":0,"1954":0,"1955":0,"1956":0,"1957":0,"1958":0,"1959":0,"1960":0,"1961":0,"1962":0,"1963":0,"1964":0,"1965":0,"1966":0,"1967":0,"1968":0,"1969":0,"1970":0,"1971":0,"1972":0,"1973":0,"1974":0,"1975":0,"1976":0,"1977":0,"1978":0,"1979":0,"1980":0,"1981":0,"1982":0,"1983":0,"1984":0,"1985":0,"1986":0,"1987":0,"1988":0,"1989":0,"1990":0,"1991":0,"1992":0,"1993":0,"1994":0,"1995":0,"1996":0,"1997":0,"1998":0,"1999":0,"2000":0,"2001":0,"2002":0,"2003":0,"2004":0,"2005":0,"2006":0,"2007":0,"2008":0,"2018":0,"2019":0,"2020":0,"2026":0,"2041":0,"2057":0,"2069":0,"2094":0,"2098":0,"2106":0,"2114":0,"2127":0,"2131":0,"2135":0,"2141":0}*
>
> it takes a while to attempt valid emitted batch.
>
> Can someone please tell me how to avoid wasting time on empty emits and
> speed up the process ?
>
> Thanks,
> Rajat
>
>
>
> On Mon, Apr 3, 2017 at 12:13 AM, Rajat Gangwar <ra...@gmail.com>
> wrote:
>
>> my topo is something like :
>>
>> .partitionAggregate(..)
>> .groupBy(..)
>> .persistentAggregate(..)
>>
>> Inserted 1 tuple, which got emitted with txn_id (batch_id) as 1.
>> But nothing happens after that. It just stuck and not even reaching
>> persistentAggregate.
>>
>> This is what i see in logs
>>
>>
>> [image: Inline image 1]
>>
>> I don't know what are these batch_ids which it is trying to re-emit.
>>
>>
>> any idea what could be the issue ?
>>
>> Thanks,
>> Rajat
>>
>>
>>
>>
>
Re: re-emitting phantom batches
Posted by Rajat Gangwar <ra...@gmail.com>.
This has slowed down the processing considerably.
I inserted 1 tuple which got emitted with txn-id : *2141*.
But when I check zookeeper node, currenttx is *1688*
*[zk: 10.X.X.X:2181(CONNECTED) 76] get
/storm_test/transactional/accrual_spout20/coordinator/currtx*
*1688*
and attempts in the queue :
[*zk: 10.X.X.X:2181(CONNECTED) 77] get
/storm_test/transactional/accrual_spout20/coordinator/currattempts*
*{"1689":2,"1690":1,"1691":1,"1692":1,"1693":1,"1694":1,"1695":1,"1696":1,"1697":1,"1698":1,"1699":1,"1700":1,"1701":1,"1702":1,"1703":1,"1704":1,"1705":1,"1706":1,"1707":1,"1708":1,"1709":1,"1710":1,"1711":1,"1712":1,"1713":1,"1714":1,"1715":1,"1716":1,"1717":1,"1718":1,"1719":1,"1720":1,"1721":1,"1722":1,"1723":1,"1724":1,"1725":1,"1726":1,"1727":1,"1728":1,"1729":1,"1730":1,"1731":1,"1732":1,"1733":1,"1734":1,"1735":1,"1736":1,"1737":1,"1738":1,"1739":1,"1740":1,"1741":1,"1742":1,"1743":1,"1744":1,"1745":1,"1746":1,"1747":1,"1748":1,"1749":1,"1750":1,"1751":1,"1752":1,"1753":1,"1754":1,"1755":1,"1756":1,"1757":1,"1758":1,"1759":1,"1760":1,"1761":1,"1762":1,"1763":1,"1764":1,"1765":1,"1766":1,"1767":1,"1768":1,"1769":1,"1770":1,"1771":1,"1772":1,"1773":1,"1774":1,"1775":1,"1776":1,"1777":1,"1778":1,"1779":1,"1780":1,"1781":1,"1782":1,"1783":1,"1784":1,"1785":1,"1786":1,"1787":1,"1788":1,"1789":1,"1790":1,"1791":1,"1792":1,"1793":1,"1794":1,"1795":1,"1796":1,"1797":1,"1798":1,"1799":1,"1800":1,"1801":1,"1802":1,"1803":1,"1804":1,"1805":1,"1806":1,"1807":1,"1808":1,"1809":1,"1810":1,"1811":1,"1812":1,"1813":1,"1814":1,"1815":1,"1816":1,"1817":1,"1818":1,"1819":1,"1820":1,"1821":1,"1822":0,"1823":0,"1824":0,"1825":0,"1826":0,"1827":0,"1828":0,"1829":0,"1830":0,"1831":0,"1832":0,"1833":0,"1834":0,"1835":0,"1836":0,"1837":1,"1838":0,"1839":0,"1840":0,"1841":1,"1842":0,"1843":0,"1844":0,"1845":0,"1846":0,"1847":0,"1848":0,"1849":0,"1850":0,"1851":1,"1852":0,"1853":0,"1854":0,"1855":0,"1856":0,"1857":1,"1858":0,"1859":0,"1860":0,"1861":0,"1862":0,"1863":0,"1864":0,"1865":0,"1866":0,"1867":0,"1868":0,"1869":0,"1870":0,"1871":0,"1872":0,"1873":0,"1874":0,"1875":0,"1876":0,"1877":0,"1878":0,"1879":1,"1880":1,"1881":0,"1882":1,"1883":0,"1884":0,"1885":0,"1886":0,"1887":0,"1888":1,"1889":0,"1890":0,"1891":0,"1892":1,"1893":1,"1894":0,"1895":0,"1896":0,"1897":0,"1898":0,"1899":0,"1900":1,"1901":0,"1902":0,"1903":0,"1904":0,"1905":0,"1906":0,"1907":0,"1908":0,"1909":0,"1910":0,"1911":0,"1912":0,"1913":0,"1914":0,"1915":0,"1916":0,"1917":0,"1918":0,"1919":1,"1920":0,"1921":1,"1922":0,"1923":0,"1924":0,"1925":0,"1926":0,"1927":0,"1928":0,"1929":0,"1930":0,"1931":0,"1932":0,"1933":0,"1934":0,"1935":0,"1936":0,"1937":0,"1938":0,"1939":0,"1940":0,"1941":0,"1942":0,"1943":0,"1944":0,"1945":0,"1946":0,"1947":0,"1948":0,"1949":0,"1950":0,"1951":0,"1952":0,"1953":0,"1954":0,"1955":0,"1956":0,"1957":0,"1958":0,"1959":0,"1960":0,"1961":0,"1962":0,"1963":0,"1964":0,"1965":0,"1966":0,"1967":0,"1968":0,"1969":0,"1970":0,"1971":0,"1972":0,"1973":0,"1974":0,"1975":0,"1976":0,"1977":0,"1978":0,"1979":0,"1980":0,"1981":0,"1982":0,"1983":0,"1984":0,"1985":0,"1986":0,"1987":0,"1988":0,"1989":0,"1990":0,"1991":0,"1992":0,"1993":0,"1994":0,"1995":0,"1996":0,"1997":0,"1998":0,"1999":0,"2000":0,"2001":0,"2002":0,"2003":0,"2004":0,"2005":0,"2006":0,"2007":0,"2008":0,"2018":0,"2019":0,"2020":0,"2026":0,"2041":0,"2057":0,"2069":0,"2094":0,"2098":0,"2106":0,"2114":0,"2127":0,"2131":0,"2135":0,"2141":0}*
it takes a while to attempt valid emitted batch.
Can someone please tell me how to avoid wasting time on empty emits and
speed up the process ?
Thanks,
Rajat
On Mon, Apr 3, 2017 at 12:13 AM, Rajat Gangwar <ra...@gmail.com>
wrote:
> my topo is something like :
>
> .partitionAggregate(..)
> .groupBy(..)
> .persistentAggregate(..)
>
> Inserted 1 tuple, which got emitted with txn_id (batch_id) as 1.
> But nothing happens after that. It just stuck and not even reaching
> persistentAggregate.
>
> This is what i see in logs
>
>
> [image: Inline image 1]
>
> I don't know what are these batch_ids which it is trying to re-emit.
>
>
> any idea what could be the issue ?
>
> Thanks,
> Rajat
>
>
>
>