You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@echarts.apache.org by GitBox <gi...@apache.org> on 2021/07/15 05:17:24 UTC
[GitHub] [echarts] pissang opened a new pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
pissang opened a new pull request #15355:
URL: https://github.com/apache/echarts/pull/15355
<!-- Please fill in the following information to help us review your PR more efficiently. -->
## Brief Information
This pull request is in the type of:
- [ ] bug fixing
- [ ] new feature
- [x] others
<!-- USE ONCE SENTENCE TO DESCRIBE WHAT THIS PR DOES. -->
### What does this PR do?
### Fixed issues
https://github.com/apache/echarts/issues/11907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673625959
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,79 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
Review comment:
This optimization is based on the profile result. In fact, in the case of #11907, The most memory cost thing is the storage of dimensions info. Previously we need to store 1000 dimensions in each series, which is 1 million in total. It's not only in the `List` instance. But also in the `Source` instance.
`createSource` will also parse all dimensions and store them in the `Source` instance. If we don't share them, the heap memory will increase from 20MB to 140MB and the render time will also be much longer. What makes it worse is it increases exponentially.
Sharing source may make things bit more complex. But I think it won't break the design of the dataset. And the good thing is it can be all done in the SourceManager and won't leak the abstraction to other modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675295137
##########
File path: src/data/DataStorage.ts
##########
@@ -0,0 +1,1281 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+import { assert, clone, isFunction, keys, map, reduce } from 'zrender/src/core/util';
+import {
+ Dictionary,
+ DimensionIndex,
+ DimensionName,
+ OptionDataItem,
+ ParsedValue,
+ ParsedValueNumeric
+} from '../util/types';
+import { DataProvider } from './helper/dataProvider';
+import { parseDataValue } from './helper/dataValueHelper';
+import OrdinalMeta from './OrdinalMeta';
+import { Source } from './Source';
+
+const UNDEFINED = 'undefined';
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+// Caution: MUST not use `new CtorUint32Array(arr, 0, len)`, because the Ctor of array is
+// different from the Ctor of typed array.
+export const CtorUint32Array = typeof Uint32Array === UNDEFINED ? Array : Uint32Array;
+export const CtorUint16Array = typeof Uint16Array === UNDEFINED ? Array : Uint16Array;
+export const CtorInt32Array = typeof Int32Array === UNDEFINED ? Array : Int32Array;
+export const CtorFloat64Array = typeof Float64Array === UNDEFINED ? Array : Float64Array;
+/**
+ * Multi dimensional data storage
+ */
+const dataCtors = {
+ 'float': CtorFloat64Array,
+ 'int': CtorInt32Array,
+ // Ordinal data type can be string or int
+ 'ordinal': Array,
+ 'number': Array,
+ 'time': CtorFloat64Array
+} as const;
+
+// Dim with same category can be convert between.
+const dataCtorCategory = {
+ 'float': 0, 'time': 0,
+ 'number': 1, 'int': 2,
+ 'ordinal': 3
+};
+
+export type DataStorageDimensionType = keyof typeof dataCtors;
+
+type DataTypedArray = Uint32Array | Int32Array | Uint16Array | Float64Array;
+type DataTypedArrayConstructor = typeof Uint32Array | typeof Int32Array | typeof Uint16Array | typeof Float64Array;
+type DataArrayLikeConstructor = typeof Array | DataTypedArrayConstructor;
+
+
+type DataValueChunk = ArrayLike<ParsedValue>;
+
+// If Ctx not specified, use List as Ctx
+type EachCb0 = (idx: number) => void;
+type EachCb1 = (x: ParsedValue, idx: number) => void;
+type EachCb2 = (x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb = (...args: any) => void;
+type FilterCb0 = (idx: number) => boolean;
+type FilterCb1 = (x: ParsedValue, idx: number) => boolean;
+type FilterCb = (...args: any) => boolean;
+type MapArrayCb = (...args: any) => any;
+type MapCb = (...args: any) => ParsedValue | ParsedValue[];
+
+export type DimValueGetter = (
+ this: DataStorage,
+ dataItem: any,
+ dimName: string,
+ dataIndex: number,
+ dimIndex: DimensionIndex
+) => ParsedValue;
+
+export interface DataStorageDimensionInfo {
+ type?: DataStorageDimensionType; // Default to be float.
+ name?: string;
+ /**
+ * When using category axis.
+ * Category strings will be collected and stored in ordinalMeta.categories.
+ * And storage will store the index of categories.
+ */
+ ordinalMeta?: OrdinalMeta,
+
+ /**
+ * Offset for ordinal parsing and collect
+ */
+ ordinalOffset?: number
+}
+
+let defaultDimValueGetters: {[sourceFormat: string]: DimValueGetter};
+
+function getIndicesCtor(rawCount: number): DataArrayLikeConstructor {
+ // The possible max value in this._indicies is always this._rawCount despite of filtering.
+ return rawCount > 65535 ? CtorUint32Array : CtorUint16Array;
+};
+function getInitialExtent(): [number, number] {
+ return [Infinity, -Infinity];
+};
+function cloneChunk(originalChunk: DataValueChunk): DataValueChunk {
+ const Ctor = originalChunk.constructor;
+ // Only shallow clone is enough when Array.
+ return Ctor === Array
+ ? (originalChunk as Array<ParsedValue>).slice()
+ : new (Ctor as DataTypedArrayConstructor)(originalChunk as DataTypedArray);
+}
+
+function prepareStorage(
+ storage: DataValueChunk[],
+ dimIdx: number,
+ dimType: DataStorageDimensionType,
+ end: number,
+ append?: boolean
+): void {
+ const DataCtor = dataCtors[dimType || 'float'];
+
+ if (append) {
+ const oldStore = storage[dimIdx];
+ const oldLen = oldStore && oldStore.length;
+ if (!(oldLen === end)) {
+ const newStore = new DataCtor(end);
+ // The cost of the copy is probably inconsiderable
+ // within the initial chunkSize.
+ for (let j = 0; j < oldLen; j++) {
+ newStore[j] = oldStore[j];
+ }
+ storage[dimIdx] = newStore;
+ }
+ }
+ else {
+ storage[dimIdx] = new DataCtor(end);
+ }
+};
+class DataStorage {
+ private _chunks: DataValueChunk[] = [];
+
+ private _provider: DataProvider;
+
+ // Raw extent will not be cloned, but only transfered.
+ // It will not be calculated util needed.
+ private _rawExtent: [number, number][] = [];
+
+ private _extent: [number, number][] = [];
+
+ // Indices stores the indices of data subset after filtered.
+ // This data subset will be used in chart.
+ private _indices: ArrayLike<any>;
+
+ private _count: number = 0;
+ private _rawCount: number = 0;
+
+ private _dimensions: DataStorageDimensionInfo[];
+ private _dimValueGetter: DimValueGetter;
+
+ defaultDimValueGetter: DimValueGetter;
+ /**
+ * Initialize from data
+ * @param data source or data or data provider.
+ */
+ initData(
+ provider: DataProvider,
+ dimensions: DataStorageDimensionInfo[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ if (__DEV__) {
+ assert(
+ isFunction(provider.getItem) && isFunction(provider.count),
+ 'Inavlid data provider.'
+ );
+ }
+
+ this._provider = provider;
+
+ // Clear
+ this._chunks = [];
+ this._indices = null;
+ this.getRawIndex = this._getRawIdxIdentity;
+
+ const defaultGetter = this.defaultDimValueGetter =
+ defaultDimValueGetters[provider.getSource().sourceFormat];
+ // Default dim value getter
+ this._dimValueGetter = dimValueGetter || defaultGetter;
+
+ const emptyObj = {};
+
+ // Reset raw extent.
+ this._rawExtent = [];
+ this._dimensions = map(dimensions, dim => ({
+ // Only pick these two props. Not leak other properties like orderMeta.
+ type: dim.type,
+ name: dim.name
+ }) as DataStorageDimensionInfo);
+
+ const dimensionsIdxMap: Dictionary<number> = {};
+ let prefix = '';
+
+ // Needs to add prefix if key is used in object prototype
+ for (let i = 0; i < dimensions.length; i++) {
+ const name = dimensions[i].name;
+ if ((emptyObj as any)[name] != null) {
+ prefix = '$$';
+ }
+ }
+ for (let i = 0; i < dimensions.length; i++) {
+ const dim = dimensions[i];
+ const name = dim.name;
+ dimensionsIdxMap[prefix + name] = i;
+ }
+
+ // We use different functions because it may be a hotspot code.
+ this.getDimensionIndex = prefix ? function (dim) {
+ return dimensionsIdxMap[prefix + dim];
+ } : function (dim) {
+ return dimensionsIdxMap[dim];
+ };
+
+ this._initDataFromProvider(0, provider.count());
+ }
+
+ getProvider(): DataProvider {
+ return this._provider;
+ }
+
+ getSource(): Source {
+ return this._provider.getSource();
+ }
+
+ getDimensionIndex: (dim: DimensionName) => number;
+
+ getDimensionCount() {
+ return this._dimensions.length;
+ }
+
+ collectOrdinalMeta(
+ dimIdx: number,
+ ordinalMeta: OrdinalMeta
+ ) {
+ const chunk = this._chunks[dimIdx];
+ const dim = this._dimensions[dimIdx];
+ const rawExtents = this._rawExtent;
+
+ const offset = dim.ordinalOffset || 0;
+ const len = chunk.length;
+
+ if (offset === 0) {
+ // We need to reset the rawExtent if collect is from start.
+ // Because this dimension may be guessed as number and calcuating a wrong extent.
+ rawExtents[dimIdx] = getInitialExtent();
+ }
+
+ const dimRawExtent = rawExtents[dimIdx];
+
+ // Parse from previous data offset. len may be changed after appendData
+ for (let i = offset; i < len; i++) {
+ const val = (chunk as any)[i] = ordinalMeta.parseAndCollect(chunk[i]);
+ dimRawExtent[0] = Math.min(val, dimRawExtent[0]);
+ dimRawExtent[1] = Math.max(val, dimRawExtent[1]);
+ }
+
+ dim.ordinalMeta = ordinalMeta;
+ dim.ordinalOffset = len;
+ dim.type = 'ordinal'; // Force to be ordinal
+ }
+
+ getOrdinalMeta(dimIdx: number) {
+ const dimInfo = this._dimensions[dimIdx];
+ const ordinalMeta = dimInfo.ordinalMeta;
+ return ordinalMeta;
+ }
+
+ /**
+ * If we using dataset.
+ * Dimensions types may only know when we initializing series.
+ * So we need to sync the type back to storage when initlializing SeriesData back
+ *
+ * Will return false if dimension type has been known and different from given.
+ * We need to recreate a new data storage in this case.
+ */
+ // TODO Can't sure what's frequency will this validate fail and cause datastorage recreate.
+ syncDimensionTypes(targetDims: DataStorageDimensionInfo[]) {
+ for (let i = 0; i < targetDims.length; i++) {
+ const targetDim = targetDims[i];
+ const selfDimIdx = this.getDimensionIndex(targetDim.name);
+ const selfDim = this._dimensions[selfDimIdx];
+ if (
+ !selfDim
+ // Dim type can be convert between because ctors are compatitable.
+ || dataCtorCategory[selfDim.type || 'float'] !== dataCtorCategory[targetDim.type || 'float']
+ // ordinalMeta is different. Usually being on the different axis.
+ || (selfDim.ordinalMeta && selfDim.ordinalMeta !== targetDim.ordinalMeta)
+ ) {
+ return false;
+ }
+ selfDim.type = targetDim.type;
Review comment:
It has been removed. Now this method has been changed to `canUse`, only validation check will be prefromed.
##########
File path: src/data/DataStorage.ts
##########
@@ -0,0 +1,1281 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+import { assert, clone, isFunction, keys, map, reduce } from 'zrender/src/core/util';
+import {
+ Dictionary,
+ DimensionIndex,
+ DimensionName,
+ OptionDataItem,
+ ParsedValue,
+ ParsedValueNumeric
+} from '../util/types';
+import { DataProvider } from './helper/dataProvider';
+import { parseDataValue } from './helper/dataValueHelper';
+import OrdinalMeta from './OrdinalMeta';
+import { Source } from './Source';
+
+const UNDEFINED = 'undefined';
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+// Caution: MUST not use `new CtorUint32Array(arr, 0, len)`, because the Ctor of array is
+// different from the Ctor of typed array.
+export const CtorUint32Array = typeof Uint32Array === UNDEFINED ? Array : Uint32Array;
+export const CtorUint16Array = typeof Uint16Array === UNDEFINED ? Array : Uint16Array;
+export const CtorInt32Array = typeof Int32Array === UNDEFINED ? Array : Int32Array;
+export const CtorFloat64Array = typeof Float64Array === UNDEFINED ? Array : Float64Array;
+/**
+ * Multi dimensional data storage
+ */
+const dataCtors = {
+ 'float': CtorFloat64Array,
+ 'int': CtorInt32Array,
+ // Ordinal data type can be string or int
+ 'ordinal': Array,
+ 'number': Array,
+ 'time': CtorFloat64Array
+} as const;
+
+// Dim with same category can be convert between.
+const dataCtorCategory = {
+ 'float': 0, 'time': 0,
+ 'number': 1, 'int': 2,
+ 'ordinal': 3
+};
+
+export type DataStorageDimensionType = keyof typeof dataCtors;
+
+type DataTypedArray = Uint32Array | Int32Array | Uint16Array | Float64Array;
+type DataTypedArrayConstructor = typeof Uint32Array | typeof Int32Array | typeof Uint16Array | typeof Float64Array;
+type DataArrayLikeConstructor = typeof Array | DataTypedArrayConstructor;
+
+
+type DataValueChunk = ArrayLike<ParsedValue>;
+
+// If Ctx not specified, use List as Ctx
+type EachCb0 = (idx: number) => void;
+type EachCb1 = (x: ParsedValue, idx: number) => void;
+type EachCb2 = (x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb = (...args: any) => void;
+type FilterCb0 = (idx: number) => boolean;
+type FilterCb1 = (x: ParsedValue, idx: number) => boolean;
+type FilterCb = (...args: any) => boolean;
+type MapArrayCb = (...args: any) => any;
+type MapCb = (...args: any) => ParsedValue | ParsedValue[];
+
+export type DimValueGetter = (
+ this: DataStorage,
+ dataItem: any,
+ dimName: string,
+ dataIndex: number,
+ dimIndex: DimensionIndex
+) => ParsedValue;
+
+export interface DataStorageDimensionInfo {
+ type?: DataStorageDimensionType; // Default to be float.
+ name?: string;
+ /**
+ * When using category axis.
+ * Category strings will be collected and stored in ordinalMeta.categories.
+ * And storage will store the index of categories.
+ */
+ ordinalMeta?: OrdinalMeta,
+
+ /**
+ * Offset for ordinal parsing and collect
+ */
+ ordinalOffset?: number
+}
+
+let defaultDimValueGetters: {[sourceFormat: string]: DimValueGetter};
+
+function getIndicesCtor(rawCount: number): DataArrayLikeConstructor {
+ // The possible max value in this._indicies is always this._rawCount despite of filtering.
+ return rawCount > 65535 ? CtorUint32Array : CtorUint16Array;
+};
+function getInitialExtent(): [number, number] {
+ return [Infinity, -Infinity];
+};
+function cloneChunk(originalChunk: DataValueChunk): DataValueChunk {
+ const Ctor = originalChunk.constructor;
+ // Only shallow clone is enough when Array.
+ return Ctor === Array
+ ? (originalChunk as Array<ParsedValue>).slice()
+ : new (Ctor as DataTypedArrayConstructor)(originalChunk as DataTypedArray);
+}
+
+function prepareStorage(
+ storage: DataValueChunk[],
+ dimIdx: number,
+ dimType: DataStorageDimensionType,
+ end: number,
+ append?: boolean
+): void {
+ const DataCtor = dataCtors[dimType || 'float'];
+
+ if (append) {
+ const oldStore = storage[dimIdx];
+ const oldLen = oldStore && oldStore.length;
+ if (!(oldLen === end)) {
+ const newStore = new DataCtor(end);
+ // The cost of the copy is probably inconsiderable
+ // within the initial chunkSize.
+ for (let j = 0; j < oldLen; j++) {
+ newStore[j] = oldStore[j];
+ }
+ storage[dimIdx] = newStore;
+ }
+ }
+ else {
+ storage[dimIdx] = new DataCtor(end);
+ }
+};
+class DataStorage {
+ private _chunks: DataValueChunk[] = [];
+
+ private _provider: DataProvider;
+
+ // Raw extent will not be cloned, but only transfered.
+ // It will not be calculated util needed.
+ private _rawExtent: [number, number][] = [];
+
+ private _extent: [number, number][] = [];
+
+ // Indices stores the indices of data subset after filtered.
+ // This data subset will be used in chart.
+ private _indices: ArrayLike<any>;
+
+ private _count: number = 0;
+ private _rawCount: number = 0;
+
+ private _dimensions: DataStorageDimensionInfo[];
+ private _dimValueGetter: DimValueGetter;
+
+ defaultDimValueGetter: DimValueGetter;
+ /**
+ * Initialize from data
+ * @param data source or data or data provider.
+ */
+ initData(
+ provider: DataProvider,
+ dimensions: DataStorageDimensionInfo[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ if (__DEV__) {
+ assert(
+ isFunction(provider.getItem) && isFunction(provider.count),
+ 'Inavlid data provider.'
+ );
+ }
+
+ this._provider = provider;
+
+ // Clear
+ this._chunks = [];
+ this._indices = null;
+ this.getRawIndex = this._getRawIdxIdentity;
+
+ const defaultGetter = this.defaultDimValueGetter =
+ defaultDimValueGetters[provider.getSource().sourceFormat];
+ // Default dim value getter
+ this._dimValueGetter = dimValueGetter || defaultGetter;
+
+ const emptyObj = {};
+
+ // Reset raw extent.
+ this._rawExtent = [];
+ this._dimensions = map(dimensions, dim => ({
+ // Only pick these two props. Not leak other properties like orderMeta.
+ type: dim.type,
+ name: dim.name
+ }) as DataStorageDimensionInfo);
+
+ const dimensionsIdxMap: Dictionary<number> = {};
+ let prefix = '';
+
+ // Needs to add prefix if key is used in object prototype
+ for (let i = 0; i < dimensions.length; i++) {
+ const name = dimensions[i].name;
+ if ((emptyObj as any)[name] != null) {
+ prefix = '$$';
+ }
+ }
+ for (let i = 0; i < dimensions.length; i++) {
+ const dim = dimensions[i];
+ const name = dim.name;
+ dimensionsIdxMap[prefix + name] = i;
+ }
+
+ // We use different functions because it may be a hotspot code.
+ this.getDimensionIndex = prefix ? function (dim) {
+ return dimensionsIdxMap[prefix + dim];
+ } : function (dim) {
+ return dimensionsIdxMap[dim];
+ };
+
+ this._initDataFromProvider(0, provider.count());
+ }
+
+ getProvider(): DataProvider {
+ return this._provider;
+ }
+
+ getSource(): Source {
+ return this._provider.getSource();
+ }
+
+ getDimensionIndex: (dim: DimensionName) => number;
+
+ getDimensionCount() {
+ return this._dimensions.length;
+ }
+
+ collectOrdinalMeta(
+ dimIdx: number,
+ ordinalMeta: OrdinalMeta
+ ) {
+ const chunk = this._chunks[dimIdx];
+ const dim = this._dimensions[dimIdx];
+ const rawExtents = this._rawExtent;
+
+ const offset = dim.ordinalOffset || 0;
+ const len = chunk.length;
+
+ if (offset === 0) {
+ // We need to reset the rawExtent if collect is from start.
+ // Because this dimension may be guessed as number and calcuating a wrong extent.
+ rawExtents[dimIdx] = getInitialExtent();
+ }
+
+ const dimRawExtent = rawExtents[dimIdx];
+
+ // Parse from previous data offset. len may be changed after appendData
+ for (let i = offset; i < len; i++) {
+ const val = (chunk as any)[i] = ordinalMeta.parseAndCollect(chunk[i]);
+ dimRawExtent[0] = Math.min(val, dimRawExtent[0]);
+ dimRawExtent[1] = Math.max(val, dimRawExtent[1]);
+ }
+
+ dim.ordinalMeta = ordinalMeta;
+ dim.ordinalOffset = len;
+ dim.type = 'ordinal'; // Force to be ordinal
+ }
+
+ getOrdinalMeta(dimIdx: number) {
+ const dimInfo = this._dimensions[dimIdx];
+ const ordinalMeta = dimInfo.ordinalMeta;
+ return ordinalMeta;
+ }
+
+ /**
+ * If we using dataset.
+ * Dimensions types may only know when we initializing series.
+ * So we need to sync the type back to storage when initlializing SeriesData back
+ *
+ * Will return false if dimension type has been known and different from given.
+ * We need to recreate a new data storage in this case.
+ */
+ // TODO Can't sure what's frequency will this validate fail and cause datastorage recreate.
+ syncDimensionTypes(targetDims: DataStorageDimensionInfo[]) {
+ for (let i = 0; i < targetDims.length; i++) {
+ const targetDim = targetDims[i];
+ const selfDimIdx = this.getDimensionIndex(targetDim.name);
+ const selfDim = this._dimensions[selfDimIdx];
+ if (
+ !selfDim
+ // Dim type can be convert between because ctors are compatitable.
+ || dataCtorCategory[selfDim.type || 'float'] !== dataCtorCategory[targetDim.type || 'float']
+ // ordinalMeta is different. Usually being on the different axis.
+ || (selfDim.ordinalMeta && selfDim.ordinalMeta !== targetDim.ordinalMeta)
+ ) {
+ return false;
+ }
+ selfDim.type = targetDim.type;
Review comment:
It has been removed. Now this method has been changed to `canUse`, only validation checks will be performed.
##########
File path: src/data/DataStorage.ts
##########
@@ -0,0 +1,1281 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+import { assert, clone, isFunction, keys, map, reduce } from 'zrender/src/core/util';
+import {
+ Dictionary,
+ DimensionIndex,
+ DimensionName,
+ OptionDataItem,
+ ParsedValue,
+ ParsedValueNumeric
+} from '../util/types';
+import { DataProvider } from './helper/dataProvider';
+import { parseDataValue } from './helper/dataValueHelper';
+import OrdinalMeta from './OrdinalMeta';
+import { Source } from './Source';
+
+const UNDEFINED = 'undefined';
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+// Caution: MUST not use `new CtorUint32Array(arr, 0, len)`, because the Ctor of array is
+// different from the Ctor of typed array.
+export const CtorUint32Array = typeof Uint32Array === UNDEFINED ? Array : Uint32Array;
+export const CtorUint16Array = typeof Uint16Array === UNDEFINED ? Array : Uint16Array;
+export const CtorInt32Array = typeof Int32Array === UNDEFINED ? Array : Int32Array;
+export const CtorFloat64Array = typeof Float64Array === UNDEFINED ? Array : Float64Array;
+/**
+ * Multi dimensional data storage
+ */
+const dataCtors = {
+ 'float': CtorFloat64Array,
+ 'int': CtorInt32Array,
+ // Ordinal data type can be string or int
+ 'ordinal': Array,
+ 'number': Array,
+ 'time': CtorFloat64Array
+} as const;
+
+// Dim with same category can be convert between.
+const dataCtorCategory = {
+ 'float': 0, 'time': 0,
+ 'number': 1, 'int': 2,
+ 'ordinal': 3
+};
+
+export type DataStorageDimensionType = keyof typeof dataCtors;
+
+type DataTypedArray = Uint32Array | Int32Array | Uint16Array | Float64Array;
+type DataTypedArrayConstructor = typeof Uint32Array | typeof Int32Array | typeof Uint16Array | typeof Float64Array;
+type DataArrayLikeConstructor = typeof Array | DataTypedArrayConstructor;
+
+
+type DataValueChunk = ArrayLike<ParsedValue>;
+
+// If Ctx not specified, use List as Ctx
+type EachCb0 = (idx: number) => void;
+type EachCb1 = (x: ParsedValue, idx: number) => void;
+type EachCb2 = (x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb = (...args: any) => void;
+type FilterCb0 = (idx: number) => boolean;
+type FilterCb1 = (x: ParsedValue, idx: number) => boolean;
+type FilterCb = (...args: any) => boolean;
+type MapArrayCb = (...args: any) => any;
+type MapCb = (...args: any) => ParsedValue | ParsedValue[];
+
+export type DimValueGetter = (
+ this: DataStorage,
+ dataItem: any,
+ dimName: string,
+ dataIndex: number,
+ dimIndex: DimensionIndex
+) => ParsedValue;
+
+export interface DataStorageDimensionInfo {
+ type?: DataStorageDimensionType; // Default to be float.
+ name?: string;
+ /**
+ * When using category axis.
+ * Category strings will be collected and stored in ordinalMeta.categories.
+ * And storage will store the index of categories.
+ */
+ ordinalMeta?: OrdinalMeta,
+
+ /**
+ * Offset for ordinal parsing and collect
+ */
+ ordinalOffset?: number
+}
+
+let defaultDimValueGetters: {[sourceFormat: string]: DimValueGetter};
+
+function getIndicesCtor(rawCount: number): DataArrayLikeConstructor {
+ // The possible max value in this._indicies is always this._rawCount despite of filtering.
+ return rawCount > 65535 ? CtorUint32Array : CtorUint16Array;
+};
+function getInitialExtent(): [number, number] {
+ return [Infinity, -Infinity];
+};
+function cloneChunk(originalChunk: DataValueChunk): DataValueChunk {
+ const Ctor = originalChunk.constructor;
+ // Only shallow clone is enough when Array.
+ return Ctor === Array
+ ? (originalChunk as Array<ParsedValue>).slice()
+ : new (Ctor as DataTypedArrayConstructor)(originalChunk as DataTypedArray);
+}
+
+function prepareStorage(
+ storage: DataValueChunk[],
+ dimIdx: number,
+ dimType: DataStorageDimensionType,
+ end: number,
+ append?: boolean
+): void {
+ const DataCtor = dataCtors[dimType || 'float'];
+
+ if (append) {
+ const oldStore = storage[dimIdx];
+ const oldLen = oldStore && oldStore.length;
+ if (!(oldLen === end)) {
+ const newStore = new DataCtor(end);
+ // The cost of the copy is probably inconsiderable
+ // within the initial chunkSize.
+ for (let j = 0; j < oldLen; j++) {
+ newStore[j] = oldStore[j];
+ }
+ storage[dimIdx] = newStore;
+ }
+ }
+ else {
+ storage[dimIdx] = new DataCtor(end);
+ }
+};
+class DataStorage {
+ private _chunks: DataValueChunk[] = [];
+
+ private _provider: DataProvider;
+
+ // Raw extent will not be cloned, but only transfered.
Review comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675303987
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
I split the original logic in https://github.com/apache/echarts/blob/master/src/data/helper/completeDimensions.ts#L115 into two:
https://github.com/apache/echarts/pull/15355/files/3de73abe262912b928e23b6eab6ac568a67bd7d7#diff-db6362f78d46bd7ee44e35f6bc622f7f7aa6d0a25ec5808f3bf481fbe7a80bb0R160
and
https://github.com/apache/echarts/pull/15355/files/3de73abe262912b928e23b6eab6ac568a67bd7d7#diff-db6362f78d46bd7ee44e35f6bc622f7f7aa6d0a25ec5808f3bf481fbe7a80bb0R139
Because I need to create and set the result dimension only when it's used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And always use `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like (1) dimensionDefineList, (2) whether dimension omitted, (3) source for this series, (4) ...).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source for "omitted dimension". If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675506703
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
Not sure why but seems this test case works as 5.1.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675130549
##########
File path: src/data/SeriesData.ts
##########
@@ -0,0 +1,1338 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+/**
+ * List for data storage
+ */
+
+import * as zrUtil from 'zrender/src/core/util';
+import {PathStyleProps} from 'zrender/src/graphic/Path';
+import Model from '../model/Model';
+import DataDiffer from './DataDiffer';
+import {DataProvider, DefaultDataProvider} from './helper/dataProvider';
+import {summarizeDimensions, DimensionSummary} from './helper/dimensionHelper';
+import DataDimensionInfo from './DataDimensionInfo';
+import {ArrayLike, Dictionary, FunctionPropertyNames} from 'zrender/src/core/types';
+import Element from 'zrender/src/Element';
+import {
+ DimensionIndex, DimensionName, DimensionLoose, OptionDataItem,
+ ParsedValue, ParsedValueNumeric, DimensionUserOuput,
+ ModelOption, SeriesDataType, OptionSourceData, SOURCE_FORMAT_TYPED_ARRAY, SOURCE_FORMAT_ORIGINAL,
+ DecalObject,
+ OrdinalNumber
+} from '../util/types';
+import {convertOptionIdName, isDataItemOption} from '../util/model';
+import { setCommonECData } from '../util/innerStore';
+import type Graph from './Graph';
+import type Tree from './Tree';
+import type { VisualMeta } from '../component/visualMap/VisualMapModel';
+import {isSourceInstance, Source} from './Source';
+import { LineStyleProps } from '../model/mixin/lineStyle';
+import DataStorage, { DimValueGetter } from './DataStorage';
+
+const isObject = zrUtil.isObject;
+const map = zrUtil.map;
+
+const CtorInt32Array = typeof Int32Array === 'undefined' ? Array : Int32Array;
+
+// Use prefix to avoid index to be the same as otherIdList[idx],
+// which will cause weird udpate animation.
+const ID_PREFIX = 'e\0\0';
+
+const INDEX_NOT_FOUND = -1;
+
+type NameRepeatCount = {[name: string]: number};
+type ItrParamDims = DimensionLoose | Array<DimensionLoose>;
+// If Ctx not specified, use List as Ctx
+type CtxOrList<Ctx> = unknown extends Ctx ? SeriesData : Ctx;
+type EachCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => void;
+type EachCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => void;
+type EachCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => void;
+type FilterCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => boolean;
+type FilterCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => boolean;
+type FilterCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => boolean;
+type FilterCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => boolean;
+type MapArrayCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => any;
+type MapArrayCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => any;
+type MapArrayCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => any;
+type MapArrayCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => any;
+type MapCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => ParsedValue | ParsedValue[];
+type MapCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) =>
+ ParsedValue | ParsedValue[];
+type MapCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => ParsedValue | ParsedValue[];
+
+const TRANSFERABLE_PROPERTIES = [
+ 'hasItemOption', '_nameList', '_idList', '_invertedIndicesMap',
+ '_rawData', '_dimValueGetter',
+ '_nameDimIdx', '_idDimIdx', '_nameRepeatCount'
+];
+const CLONE_PROPERTIES = [
+ '_approximateExtent'
+];
+
+export interface DefaultDataVisual {
+ style: PathStyleProps
+ // Draw type determined which prop should be set with encoded color.
+ // It's only available on the global visual. Use getVisual('drawType') to access it.
+ // It will be set in visual/style.ts module in the first priority.
+ drawType: 'fill' | 'stroke'
+
+ symbol?: string
+ symbolSize?: number | number[]
+ symbolRotate?: number
+ symbolKeepAspect?: boolean
+ symbolOffset?: string | number | (string | number)[]
+
+ liftZ?: number
+ // For legend.
+ legendIcon?: string
+ legendLineStyle?: LineStyleProps
+
+ // visualMap will inject visualMeta data
+ visualMeta?: VisualMeta[]
+
+ // If color is encoded from palette
+ colorFromPalette?: boolean
+
+ decal?: DecalObject
+}
+
+export interface DataCalculationInfo<SERIES_MODEL> {
+ stackedDimension: string;
+ stackedByDimension: string;
+ isStackedByIndex: boolean;
+ stackedOverDimension: string;
+ stackResultDimension: string;
+ stackedOnSeries?: SERIES_MODEL;
+}
+
+// -----------------------------
+// Internal method declarations:
+// -----------------------------
+let prepareInvertedIndex: (data: SeriesData) => void;
+let getId: (data: SeriesData, rawIndex: number) => string;
+let getIdNameFromStore: (data: SeriesData, dimIdx: number, dataIdx: number) => string;
+let normalizeDimensions: (dimensions: ItrParamDims) => Array<DimensionLoose>;
+let validateDimensions: (data: SeriesData, dims: DimensionIndex[]) => void;
+let transferProperties: (target: SeriesData, source: SeriesData) => void;
+let cloneListForMapAndSample: (original: SeriesData) => SeriesData;
+let makeIdFromName: (data: SeriesData, idx: number) => void;
+
+class SeriesData<
+ HostModel extends Model = Model,
+ Visual extends DefaultDataVisual = DefaultDataVisual
+> {
+
+ readonly type = 'list';
+
+ readonly dimensions: string[];
+
+ // Infomation of each data dimension, like data type.
+ private _dimensionInfos: {[dimName: string]: DataDimensionInfo};
+
+ readonly hostModel: HostModel;
+
+ /**
+ * @readonly
+ */
+ dataType: SeriesDataType;
+
+ /**
+ * @readonly
+ * Host graph if List is used to store graph nodes / edges.
+ */
+ graph?: Graph;
+
+ /**
+ * @readonly
+ * Host tree if List is used to store tree ndoes.
+ */
+ tree?: Tree;
+
+ private _store: DataStorage;
+
+ private _nameList: string[] = [];
+ private _idList: string[] = [];
+
+ // Models of data option is stored sparse for optimizing memory cost
+ // Never used yet (not used yet).
+ // private _optionModels: Model[] = [];
+
+ // Global visual properties after visual coding
+ private _visual: Dictionary<any> = {};
+
+ // Globel layout properties.
+ private _layout: Dictionary<any> = {};
+
+ // Item visual properties after visual coding
+ private _itemVisuals: Dictionary<any>[] = [];
+
+ // Item layout properties after layout
+ private _itemLayouts: any[] = [];
+
+ // Graphic elemnents
+ private _graphicEls: Element[] = [];
+
+ // key: dim, value: extent
+ private _approximateExtent: {[dimName: string]: [number, number]} = {};
+
+ private _dimensionsSummary: DimensionSummary;
+
+ private _invertedIndicesMap: {[dimName: string]: ArrayLike<number>};
+
+ private _calculationInfo: DataCalculationInfo<HostModel> = {} as DataCalculationInfo<HostModel>;
+
+ // User output info of this data.
+ // DO NOT use it in other places!
+ // When preparing user params for user callbacks, we have
+ // to clone these inner data structures to prevent users
+ // from modifying them to effect built-in logic. And for
+ // performance consideration we make this `userOutput` to
+ // avoid clone them too many times.
+ readonly userOutput: DimensionUserOuput;
+
+ // Having detected that there is data item is non primitive type
+ // (in type `OptionDataItemObject`).
+ // Like `data: [ { value: xx, itemStyle: {...} }, ...]`
+ // At present it only happen in `SOURCE_FORMAT_ORIGINAL`.
+ hasItemOption: boolean = false;
+
+ // id or name is used on dynamic data, mapping old and new items.
+ // When generating id from name, avoid repeat.
+ private _nameRepeatCount: NameRepeatCount;
+ private _nameDimIdx: number;
+ private _idDimIdx: number;
+
+ private __wrappedMethods: string[];
+
+ // Methods that create a new list based on this list should be listed here.
+ // Notice that those method should `RETURN` the new list.
+ TRANSFERABLE_METHODS = ['cloneShallow', 'downSample', 'lttbDownSample', 'map'] as const;
+ // Methods that change indices of this list should be listed here.
+ CHANGABLE_METHODS = ['filterSelf', 'selectRange'] as const;
+ DOWNSAMPLE_METHODS = ['downSample', 'lttbDownSample'] as const;
+
+ /**
+ * @param dimensions
+ * For example, ['someDimName', {name: 'someDimName', type: 'someDimType'}, ...].
+ * Dimensions should be concrete names like x, y, z, lng, lat, angle, radius
+ */
+ constructor(dimensions: Array<string | object | DataDimensionInfo>, hostModel: HostModel) {
+ dimensions = dimensions || ['x', 'y'];
+
+ const dimensionInfos: Dictionary<DataDimensionInfo> = {};
+ const dimensionNames = [];
+ const invertedIndicesMap: Dictionary<number[]> = {};
+
+ for (let i = 0; i < dimensions.length; i++) {
+ // Use the original dimensions[i], where other flag props may exists.
+ const dimInfoInput = dimensions[i];
+
+ const dimensionInfo: DataDimensionInfo =
+ zrUtil.isString(dimInfoInput)
+ ? new DataDimensionInfo({name: dimInfoInput})
+ : !(dimInfoInput instanceof DataDimensionInfo)
+ ? new DataDimensionInfo(dimInfoInput)
+ : dimInfoInput;
+
+ const dimensionName = dimensionInfo.name;
+ dimensionInfo.type = dimensionInfo.type || 'float';
+ if (!dimensionInfo.coordDim) {
+ dimensionInfo.coordDim = dimensionName;
+ dimensionInfo.coordDimIndex = 0;
+ }
+
+ const otherDims = dimensionInfo.otherDims = dimensionInfo.otherDims || {};
+ dimensionNames.push(dimensionName);
+ dimensionInfos[dimensionName] = dimensionInfo;
+
+ dimensionInfo.index = i;
+
+ if (dimensionInfo.createInvertedIndices) {
+ invertedIndicesMap[dimensionName] = [];
+ }
+ if (otherDims.itemName === 0) {
+ this._nameDimIdx = i;
+ }
+ if (otherDims.itemId === 0) {
+ this._idDimIdx = i;
+ }
+ }
+
+ this.dimensions = dimensionNames;
+ this._dimensionInfos = dimensionInfos;
+ this.hostModel = hostModel;
+
+ // Cache summary info for fast visit. See "dimensionHelper".
+ this._dimensionsSummary = summarizeDimensions(this);
+
+ this._invertedIndicesMap = invertedIndicesMap;
+
+ this.userOutput = this._dimensionsSummary.userOutput;
+ }
+
+ /**
+ * The meanings of the input parameter `dim`:
+ *
+ * + If dim is a number (e.g., `1`), it means the index of the dimension.
+ * For example, `getDimension(0)` will return 'x' or 'lng' or 'radius'.
+ * + If dim is a number-like string (e.g., `"1"`):
+ * + If there is the same concrete dim name defined in `this.dimensions`, it means that concrete name.
+ * + If not, it will be converted to a number, which means the index of the dimension.
+ * (why? because of the backward compatbility. We have been tolerating number-like string in
+ * dimension setting, although now it seems that it is not a good idea.)
+ * For example, `visualMap[i].dimension: "1"` is the same meaning as `visualMap[i].dimension: 1`,
+ * if no dimension name is defined as `"1"`.
+ * + If dim is a not-number-like string, it means the concrete dim name.
+ * For example, it can be be default name `"x"`, `"y"`, `"z"`, `"lng"`, `"lat"`, `"angle"`, `"radius"`,
+ * or customized in `dimensions` property of option like `"age"`.
+ *
+ * Get dimension name
+ * @param dim See above.
+ * @return Concrete dim name.
+ */
+ getDimension(dim: DimensionLoose): DimensionName {
+ if (typeof dim === 'number'
+ // If being a number-like string but not being defined a dimension name.
+ || (!isNaN(dim as any) && !this._dimensionInfos.hasOwnProperty(dim))
+ ) {
+ dim = this.dimensions[dim as DimensionIndex];
+ }
+ return dim as DimensionName;
+ }
+
+ private _getStoreDimIndex(dim: DimensionLoose): DimensionIndex {
+ return this._store.getDimensionIndex(this.getDimension(dim));
+ }
+
+ /**
+ * Get type and calculation info of particular dimension
+ * @param dim
+ * Dimension can be concrete names like x, y, z, lng, lat, angle, radius
+ * Or a ordinal number. For example getDimensionInfo(0) will return 'x' or 'lng' or 'radius'
+ */
+ getDimensionInfo(dim: DimensionLoose): DataDimensionInfo {
+ // Do not clone, because there may be categories in dimInfo.
+ return this._dimensionInfos[this.getDimension(dim)];
+ }
+
+ /**
+ * concrete dimension name list on coord.
+ */
+ getDimensionsOnCoord(): DimensionName[] {
+ return this._dimensionsSummary.dataDimsOnCoord.slice();
+ }
+
+ /**
+ * @param coordDim
+ * @param idx A coordDim may map to more than one data dim.
+ * If not specified, return the first dim not extra.
+ * @return concrete data dim. If not found, return null/undefined
+ */
+ mapDimension(coordDim: DimensionName): DimensionName;
+ mapDimension(coordDim: DimensionName, idx: number): DimensionName;
+ mapDimension(coordDim: DimensionName, idx?: number): DimensionName {
+ const dimensionsSummary = this._dimensionsSummary;
+
+ if (idx == null) {
+ return dimensionsSummary.encodeFirstDimNotExtra[coordDim] as any;
+ }
+
+ const dims = dimensionsSummary.encode[coordDim];
+ return dims ? dims[idx as number] as any : null;
+ }
+
+ mapDimensionsAll(coordDim: DimensionName): DimensionName[] {
+ const dimensionsSummary = this._dimensionsSummary;
+ const dims = dimensionsSummary.encode[coordDim];
+ return (dims || []).slice();
+ }
+
+ getStorage() {
+ return this._store;
+ }
+
+ /**
+ * Initialize from data
+ * @param data source or data or data storage.
+ * @param nameList The name of a datum is used on data diff and
+ * default label/tooltip.
+ * A name can be specified in encode.itemName,
+ * or dataItem.name (only for series option data),
+ * or provided in nameList from outside.
+ */
+ initData(
+ data: Source | OptionSourceData | DataStorage | DataProvider,
+ nameList?: string[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ let store: DataStorage;
+ const dimensions = this.dimensions;
+ const dimensionInfos = map(dimensions, dimName => this._dimensionInfos[dimName]);
+ if (data instanceof DataStorage) {
+ if (data.syncDimensionTypes(dimensionInfos)) {
+ store = data;
+ }
+ // Sync failed
+ else {
+ // Needs to recreate data storage if it's not match the given dimension.
+ data = data.getSource();
+ }
+ }
+
+ if (!store) {
+ const provider = (isSourceInstance(data) || zrUtil.isArrayLike(data))
+ ? new DefaultDataProvider(data as Source | OptionSourceData, dimensions.length)
+ : data as DataProvider;
+ store = new DataStorage();
+ store.initData(provider, dimensionInfos, dimValueGetter);
+ }
+
+ this._store = store;
+
+ // Reset
+ this._nameList = (nameList || []).slice();
+ this._idList = [];
+ this._nameRepeatCount = {};
+
+ this._doInit(0, store.count());
+ }
+
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ */
+ appendData(data: ArrayLike<any>): void {
+ const range = this._store.appendData(data);
+ this._doInit(range[0], range[1]);
+ }
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ * This method does not modify `rawData` (`dataProvider`), but only
+ * add values to storage.
+ *
+ * The final count will be increased by `Math.max(values.length, names.length)`.
+ *
+ * @param values That is the SourceType: 'arrayRows', like
+ * [
+ * [12, 33, 44],
+ * [NaN, 43, 1],
+ * ['-', 'asdf', 0]
+ * ]
+ * Each item is exaclty cooresponding to a dimension.
+ */
+ appendValues(values: any[][], names?: string[]): void {
+ const {start, end} = this._store.appendValues(values, names.length);
+ const shouldMakeIdFromName = this._shouldMakeIdFromName();
+
+ this._updateOrdinalMeta();
+
+ if (names) {
+ for (let idx = start; idx < end; idx++) {
+ const sourceIdx = idx - start;
+ this._nameList[idx] = names[sourceIdx];
+ if (shouldMakeIdFromName) {
+ makeIdFromName(this, idx);
+ }
+ }
+ }
+ }
+
+ private _updateOrdinalMeta() {
+ const store = this._store;
+ const dimensions = this.dimensions;
+ for (let i = 0; i < dimensions.length; i++) {
+ const dimInfo = this._dimensionInfos[dimensions[i]];
+ if (dimInfo.ordinalMeta) {
+ const dimIdx = store.getDimensionIndex(dimensions[i]);
+ store.collectOrdinalMeta(dimIdx, dimInfo.ordinalMeta);
+ }
+ }
+ }
+
+ private _shouldMakeIdFromName() {
+ const provider = this._store.getProvider();
+ return this._idDimIdx == null
+ && provider.getSource().sourceFormat !== SOURCE_FORMAT_TYPED_ARRAY
+ && !provider.fillStorage;
+ }
+
+ private _doInit(start: number, end: number): void {
+ if (start >= end) {
+ return;
+ }
+
+ const store = this._store;
+ const provider = store.getProvider();
+
+ this._updateOrdinalMeta();
+
+ const nameList = this._nameList;
+ const idList = this._idList;
+ const sourceFormat = provider.getSource().sourceFormat;
+ const isFormatOriginal = sourceFormat === SOURCE_FORMAT_ORIGINAL;
+
+ // Each data item is value
+ // [1, 2]
+ // 2
+ // Bar chart, line chart which uses category axis
+ // only gives the 'y' value. 'x' value is the indices of category
+ // Use a tempValue to normalize the value to be a (x, y) value
+ // If dataItem is {name: ...} or {id: ...}, it has highest priority.
+ // This kind of ids and names are always stored `_nameList` and `_idList`.
+ if (isFormatOriginal && !provider.pure) {
+ const sharedDataItem = [] as OptionDataItem;
+ for (let idx = start; idx < end; idx++) {
+ // NOTICE: Try not to write things into dataItem
+ const dataItem = provider.getItem(idx, sharedDataItem);
+ if (!this.hasItemOption && isDataItemOption(dataItem)) {
+ this.hasItemOption = true;
+ }
+ if (dataItem) {
+ const itemName = (dataItem as any).name;
+ if (nameList[idx] == null && itemName != null) {
+ nameList[idx] = convertOptionIdName(itemName, null);
+ }
+ const itemId = (dataItem as any).id;
+ if (idList[idx] == null && itemId != null) {
+ idList[idx] = convertOptionIdName(itemId, null);
+ }
+ }
+ }
+ }
+
+ if (this._shouldMakeIdFromName()) {
+ for (let idx = start; idx < end; idx++) {
+ makeIdFromName(this, idx);
+ }
+ }
+
+ prepareInvertedIndex(this);
+ }
+ /**
+ * PENDING: In fact currently this function is only used to short-circuit
+ * the calling of `scale.unionExtentFromData` when data have been filtered by modules
+ * like "dataZoom". `scale.unionExtentFromData` is used to calculate data extent for series on
+ * an axis, but if a "axis related data filter module" is used, the extent of the axis have
+ * been fixed and no need to calling `scale.unionExtentFromData` actually.
+ * But if we add "custom data filter" in future, which is not "axis related", this method may
+ * be still needed.
+ *
+ * Optimize for the scenario that data is filtered by a given extent.
+ * Consider that if data amount is more than hundreds of thousand,
+ * extent calculation will cost more than 10ms and the cache will
+ * be erased because of the filtering.
+ */
+ getApproximateExtent(dim: DimensionLoose): [number, number] {
+ return this._approximateExtent[dim] || this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ /**
+ * Calculate extent on a filtered data might be time consuming.
+ * Approximate extent is only used for: calculte extent of filtered data outside.
+ */
+ setApproximateExtent(extent: [number, number], dim: DimensionLoose): void {
+ dim = this.getDimension(dim);
+ this._approximateExtent[dim] = extent.slice() as [number, number];
+ }
+
+ getCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY
+ ): DataCalculationInfo<HostModel>[CALC_INFO_KEY] {
+ return this._calculationInfo[key];
+ }
+
+ /**
+ * @param key or k-v object
+ */
+ setCalculationInfo(
+ key: DataCalculationInfo<HostModel>
+ ): void;
+ setCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY,
+ value: DataCalculationInfo<HostModel>[CALC_INFO_KEY]
+ ): void;
+ setCalculationInfo(
+ key: (keyof DataCalculationInfo<HostModel>) | DataCalculationInfo<HostModel>,
+ value?: DataCalculationInfo<HostModel>[keyof DataCalculationInfo<HostModel>]
+ ): void {
+ isObject(key)
+ ? zrUtil.extend(this._calculationInfo, key as object)
+ : ((this._calculationInfo as any)[key] = value);
+ }
+
+ /**
+ * @return Never be null/undefined. `number` will be converted to string. Becuase:
+ * In most cases, name is used in display, where returning a string is more convenient.
+ * In other cases, name is used in query (see `indexOfName`), where we can keep the
+ * rule that name `2` equals to name `'2'`.
+ */
+ getName(idx: number): string {
+ const rawIndex = this.getRawIndex(idx);
+ let name = this._nameList[rawIndex];
+ if (name == null && this._nameDimIdx != null) {
+ name = getIdNameFromStore(this, this._nameDimIdx, rawIndex);
+ }
+ if (name == null) {
+ name = '';
+ }
+ return name;
+ }
+
+ private _getCategory(dimIdx: number, idx: number) {
+ const ordinal = this._store.get(dimIdx, idx);
+ const ordinalMeta = this._store.getOrdinalMeta(dimIdx);
+ if (ordinalMeta) {
+ return ordinalMeta.categories[ordinal as OrdinalNumber];
+ }
+ return ordinal;
+ }
+
+ /**
+ * @return Never null/undefined. `number` will be converted to string. Becuase:
+ * In all cases having encountered at present, id is used in making diff comparison, which
+ * are usually based on hash map. We can keep the rule that the internal id are always string
+ * (treat `2` is the same as `'2'`) to make the related logic simple.
+ */
+ getId(idx: number): string {
+ return getId(this, this.getRawIndex(idx));
+ }
+
+ count() {
+ return this._store.count();
+ }
+ /**
+ * Get value. Return NaN if idx is out of range.
+ * @param dim Dim must be concrete name.
+ */
+ get(dim: DimensionName, idx: number): ParsedValue {
+ const store = this._store;
+ return store.get(store.getDimensionIndex(dim), idx);
+ }
+
+ getByRawIndex(dim: DimensionName, rawIdx: number): ParsedValue {
+ const store = this._store;
+ return store.getByRawIndex(store.getDimensionIndex(dim), rawIdx);
+ }
+
+ getIndices() {
+ return this._store.getIndices();
+ }
+
+ getDataExtent(dim: DimensionLoose): [number, number] {
+ return this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ getSum(dim: DimensionLoose): number {
+ return this._store.getSum(this._getStoreDimIndex(dim));
+ }
+
+ getMedian(dim: DimensionLoose): number {
+ return this._store.getMedian(this._getStoreDimIndex(dim));
+ }
+ /**
+ * Get value for multi dimensions.
+ * @param dimensions If ignored, using all dimensions.
+ */
+ getValues(idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[], idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[] | number, idx?: number): ParsedValue[] {
+ const store = this._store;
+ return zrUtil.isArray(dimensions)
+ ? store.getValues(map(dimensions, dim => this._getStoreDimIndex(dim)), idx)
+ : store.getValues(dimensions as number);
+ }
+
+ /**
+ * If value is NaN. Inlcuding '-'
+ * Only check the coord dimensions.
+ */
+ hasValue(idx: number): boolean {
+ const dataDimsOnCoord = this._dimensionsSummary.dataDimsOnCoord;
+ for (let i = 0, len = dataDimsOnCoord.length; i < len; i++) {
+ // Ordinal type originally can be string or number.
+ // But when an ordinal type is used on coord, it can
+ // not be string but only number. So we can also use isNaN.
+ if (isNaN(this.get(dataDimsOnCoord[i], idx) as any)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Retreive the index with given name
+ */
+ indexOfName(name: string): number {
+ for (let i = 0, len = this._store.count(); i < len; i++) {
+ if (this.getName(i) === name) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ getRawIndex(idx: number) {
+ return this._store.getRawIndex(idx);
+ }
+
+ indexOfRawIndex(rawIndex: number) {
+ return this._store.indexOfRawIndex(rawIndex);
+ }
+
+ /**
+ * Only support the dimension which inverted index created.
+ * Do not support other cases until required.
+ * @param dim concrete dim
+ * @param value ordinal index
+ * @return rawIndex
+ */
+ rawIndexOf(dim: DimensionName, value: OrdinalNumber): number {
+ const invertedIndices = dim && this._invertedIndicesMap[dim];
+ if (__DEV__) {
+ if (!invertedIndices) {
+ throw new Error('Do not supported yet');
+ }
+ }
+ const rawIndex = invertedIndices[value];
+ if (rawIndex == null || isNaN(rawIndex)) {
+ return INDEX_NOT_FOUND;
+ }
+ return rawIndex;
+ }
+
+ /**
+ * Retreive the index of nearest value
+ * @param dim
+ * @param value
+ * @param [maxDistance=Infinity]
+ * @return If and only if multiple indices has
+ * the same value, they are put to the result.
+ */
+ indicesOfNearest(dim: DimensionLoose, value: number, maxDistance?: number): number[] {
+ return this._store.indicesOfNearest(
+ this._getStoreDimIndex(dim),
+ value, maxDistance
+ );
+ }
+ /**
+ * Data iteration
+ * @param ctx default this
+ * @example
+ * list.each('x', function (x, idx) {});
+ * list.each(['x', 'y'], function (x, y, idx) {});
+ * list.each(function (idx) {})
+ */
+ each<Ctx>(cb: EachCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): void;
+ each<Ctx>(dims: DimensionLoose, cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose], cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: EachCb2<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: ItrParamDims, cb: EachCb<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(
+ dims: ItrParamDims | EachCb<Ctx>,
+ cb: EachCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): void {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store.each(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ }
+ /**
+ * Data filter
+ */
+ filterSelf<Ctx>(cb: FilterCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): this;
+ filterSelf<Ctx>(dims: DimensionLoose, cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose], cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: FilterCb2<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: ItrParamDims, cb: FilterCb<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(
+ dims: ItrParamDims | FilterCb<Ctx>,
+ cb: FilterCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ // Clone first
+ this._store = this._store.clone();
+ this._store.filterSelf(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+
+ return this;
+ }
+
+ /**
+ * Select data in range. (For optimization of filter)
+ * (Manually inline code, support 5 million data filtering in data zoom.)
+ */
+ selectRange(range: Record<string, [number, number]>): SeriesData {
+ 'use strict';
+
+ const innerRange: Record<number, [number, number]> = {};
+ const dims = zrUtil.keys(range);
+ const dimIndices: number[] = [];
+ zrUtil.each(dims, (dim) => {
+ const dimIdx = this._getStoreDimIndex(dim);
+ innerRange[dimIdx] = range[dim];
+ dimIndices.push(dimIdx);
+ });
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store = this._store.clone();
+ this._store.selectRange(innerRange);
+ return this;
+ }
+
+ /**
+ * Data mapping to a plain array
+ */
+ mapArray<Ctx, Cb extends MapArrayCb0<Ctx>>(cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-disable */
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: DimensionLoose, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: [DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb2<Ctx>>(dims: [DimensionLoose, DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb<Ctx>>(dims: ItrParamDims, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-enable */
+ mapArray<Ctx>(
+ dims: ItrParamDims | MapArrayCb<Ctx>,
+ cb: MapArrayCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): unknown[] {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ ctx = (ctx || this) as Ctx;
+
+ const result: unknown[] = [];
+ this.each(dims, function () {
+ result.push(cb && (cb as MapArrayCb<Ctx>).apply(this, arguments));
+ }, ctx);
+ return result;
+ }
+
+ /**
+ * Data mapping to a new List with given dimensions
+ */
+ map<Ctx>(dims: DimensionLoose, cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(dims: [DimensionLoose], cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ /* eslint-disable-next-line */
+ map<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: MapCb2<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(
+ dims: ItrParamDims,
+ cb: MapCb<Ctx>,
+ ctx?: Ctx,
+ ctxCompat?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || ctxCompat || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(
+ normalizeDimensions(dims), this._getStoreDimIndex, this
+ );
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ const list = cloneListForMapAndSample(this);
+ list._store = this._store.map(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ return list;
+ }
+
+ /**
+ * Large data down sampling on given dimension
+ * @param sampleIndex Sample index for name and id
+ */
+ downSample(
Review comment:
Yes I agree, `DataStorage` will be totally immutable. (The only exception is `appendData` and `appendValue`, which are restricted to only be called in `Series#getRawData`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675393751
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
I add default dimensions generation in the dataset. It seems more reasonable than picking the dimensions from the coordinate system from the dataset side. But for better display in the tooltip. it will still use `displayName` from coordinate system. Which is 'x', 'y', 'value', etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r674490745
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
> Do we need to distinguish user defined string 'null' or 'undefined' from null / undefined here?
I don't understand what you mean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r682374253
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
@100pah Fixed it. The original `SeriesData#getDimensionInfo().index` is replaced by `SeriesData#getDimensionIndex`.
Which is always correct for reading `SeriesData#getRawValue`.
Currently the relationship between `SeriesData#dimensions` and `SeriesData#getStorageDimensions` is a bit confusing and I commented to explain it. But still not sure if we will use it wrong further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
5. If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same data. And then a data storage created by based on s
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And alway user `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like dimensionDefineList, whether dimension omitted, source for this series).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source for "omitted dimension". If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675130549
##########
File path: src/data/SeriesData.ts
##########
@@ -0,0 +1,1338 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+/**
+ * List for data storage
+ */
+
+import * as zrUtil from 'zrender/src/core/util';
+import {PathStyleProps} from 'zrender/src/graphic/Path';
+import Model from '../model/Model';
+import DataDiffer from './DataDiffer';
+import {DataProvider, DefaultDataProvider} from './helper/dataProvider';
+import {summarizeDimensions, DimensionSummary} from './helper/dimensionHelper';
+import DataDimensionInfo from './DataDimensionInfo';
+import {ArrayLike, Dictionary, FunctionPropertyNames} from 'zrender/src/core/types';
+import Element from 'zrender/src/Element';
+import {
+ DimensionIndex, DimensionName, DimensionLoose, OptionDataItem,
+ ParsedValue, ParsedValueNumeric, DimensionUserOuput,
+ ModelOption, SeriesDataType, OptionSourceData, SOURCE_FORMAT_TYPED_ARRAY, SOURCE_FORMAT_ORIGINAL,
+ DecalObject,
+ OrdinalNumber
+} from '../util/types';
+import {convertOptionIdName, isDataItemOption} from '../util/model';
+import { setCommonECData } from '../util/innerStore';
+import type Graph from './Graph';
+import type Tree from './Tree';
+import type { VisualMeta } from '../component/visualMap/VisualMapModel';
+import {isSourceInstance, Source} from './Source';
+import { LineStyleProps } from '../model/mixin/lineStyle';
+import DataStorage, { DimValueGetter } from './DataStorage';
+
+const isObject = zrUtil.isObject;
+const map = zrUtil.map;
+
+const CtorInt32Array = typeof Int32Array === 'undefined' ? Array : Int32Array;
+
+// Use prefix to avoid index to be the same as otherIdList[idx],
+// which will cause weird udpate animation.
+const ID_PREFIX = 'e\0\0';
+
+const INDEX_NOT_FOUND = -1;
+
+type NameRepeatCount = {[name: string]: number};
+type ItrParamDims = DimensionLoose | Array<DimensionLoose>;
+// If Ctx not specified, use List as Ctx
+type CtxOrList<Ctx> = unknown extends Ctx ? SeriesData : Ctx;
+type EachCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => void;
+type EachCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => void;
+type EachCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => void;
+type FilterCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => boolean;
+type FilterCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => boolean;
+type FilterCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => boolean;
+type FilterCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => boolean;
+type MapArrayCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => any;
+type MapArrayCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => any;
+type MapArrayCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => any;
+type MapArrayCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => any;
+type MapCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => ParsedValue | ParsedValue[];
+type MapCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) =>
+ ParsedValue | ParsedValue[];
+type MapCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => ParsedValue | ParsedValue[];
+
+const TRANSFERABLE_PROPERTIES = [
+ 'hasItemOption', '_nameList', '_idList', '_invertedIndicesMap',
+ '_rawData', '_dimValueGetter',
+ '_nameDimIdx', '_idDimIdx', '_nameRepeatCount'
+];
+const CLONE_PROPERTIES = [
+ '_approximateExtent'
+];
+
+export interface DefaultDataVisual {
+ style: PathStyleProps
+ // Draw type determined which prop should be set with encoded color.
+ // It's only available on the global visual. Use getVisual('drawType') to access it.
+ // It will be set in visual/style.ts module in the first priority.
+ drawType: 'fill' | 'stroke'
+
+ symbol?: string
+ symbolSize?: number | number[]
+ symbolRotate?: number
+ symbolKeepAspect?: boolean
+ symbolOffset?: string | number | (string | number)[]
+
+ liftZ?: number
+ // For legend.
+ legendIcon?: string
+ legendLineStyle?: LineStyleProps
+
+ // visualMap will inject visualMeta data
+ visualMeta?: VisualMeta[]
+
+ // If color is encoded from palette
+ colorFromPalette?: boolean
+
+ decal?: DecalObject
+}
+
+export interface DataCalculationInfo<SERIES_MODEL> {
+ stackedDimension: string;
+ stackedByDimension: string;
+ isStackedByIndex: boolean;
+ stackedOverDimension: string;
+ stackResultDimension: string;
+ stackedOnSeries?: SERIES_MODEL;
+}
+
+// -----------------------------
+// Internal method declarations:
+// -----------------------------
+let prepareInvertedIndex: (data: SeriesData) => void;
+let getId: (data: SeriesData, rawIndex: number) => string;
+let getIdNameFromStore: (data: SeriesData, dimIdx: number, dataIdx: number) => string;
+let normalizeDimensions: (dimensions: ItrParamDims) => Array<DimensionLoose>;
+let validateDimensions: (data: SeriesData, dims: DimensionIndex[]) => void;
+let transferProperties: (target: SeriesData, source: SeriesData) => void;
+let cloneListForMapAndSample: (original: SeriesData) => SeriesData;
+let makeIdFromName: (data: SeriesData, idx: number) => void;
+
+class SeriesData<
+ HostModel extends Model = Model,
+ Visual extends DefaultDataVisual = DefaultDataVisual
+> {
+
+ readonly type = 'list';
+
+ readonly dimensions: string[];
+
+ // Infomation of each data dimension, like data type.
+ private _dimensionInfos: {[dimName: string]: DataDimensionInfo};
+
+ readonly hostModel: HostModel;
+
+ /**
+ * @readonly
+ */
+ dataType: SeriesDataType;
+
+ /**
+ * @readonly
+ * Host graph if List is used to store graph nodes / edges.
+ */
+ graph?: Graph;
+
+ /**
+ * @readonly
+ * Host tree if List is used to store tree ndoes.
+ */
+ tree?: Tree;
+
+ private _store: DataStorage;
+
+ private _nameList: string[] = [];
+ private _idList: string[] = [];
+
+ // Models of data option is stored sparse for optimizing memory cost
+ // Never used yet (not used yet).
+ // private _optionModels: Model[] = [];
+
+ // Global visual properties after visual coding
+ private _visual: Dictionary<any> = {};
+
+ // Globel layout properties.
+ private _layout: Dictionary<any> = {};
+
+ // Item visual properties after visual coding
+ private _itemVisuals: Dictionary<any>[] = [];
+
+ // Item layout properties after layout
+ private _itemLayouts: any[] = [];
+
+ // Graphic elemnents
+ private _graphicEls: Element[] = [];
+
+ // key: dim, value: extent
+ private _approximateExtent: {[dimName: string]: [number, number]} = {};
+
+ private _dimensionsSummary: DimensionSummary;
+
+ private _invertedIndicesMap: {[dimName: string]: ArrayLike<number>};
+
+ private _calculationInfo: DataCalculationInfo<HostModel> = {} as DataCalculationInfo<HostModel>;
+
+ // User output info of this data.
+ // DO NOT use it in other places!
+ // When preparing user params for user callbacks, we have
+ // to clone these inner data structures to prevent users
+ // from modifying them to effect built-in logic. And for
+ // performance consideration we make this `userOutput` to
+ // avoid clone them too many times.
+ readonly userOutput: DimensionUserOuput;
+
+ // Having detected that there is data item is non primitive type
+ // (in type `OptionDataItemObject`).
+ // Like `data: [ { value: xx, itemStyle: {...} }, ...]`
+ // At present it only happen in `SOURCE_FORMAT_ORIGINAL`.
+ hasItemOption: boolean = false;
+
+ // id or name is used on dynamic data, mapping old and new items.
+ // When generating id from name, avoid repeat.
+ private _nameRepeatCount: NameRepeatCount;
+ private _nameDimIdx: number;
+ private _idDimIdx: number;
+
+ private __wrappedMethods: string[];
+
+ // Methods that create a new list based on this list should be listed here.
+ // Notice that those method should `RETURN` the new list.
+ TRANSFERABLE_METHODS = ['cloneShallow', 'downSample', 'lttbDownSample', 'map'] as const;
+ // Methods that change indices of this list should be listed here.
+ CHANGABLE_METHODS = ['filterSelf', 'selectRange'] as const;
+ DOWNSAMPLE_METHODS = ['downSample', 'lttbDownSample'] as const;
+
+ /**
+ * @param dimensions
+ * For example, ['someDimName', {name: 'someDimName', type: 'someDimType'}, ...].
+ * Dimensions should be concrete names like x, y, z, lng, lat, angle, radius
+ */
+ constructor(dimensions: Array<string | object | DataDimensionInfo>, hostModel: HostModel) {
+ dimensions = dimensions || ['x', 'y'];
+
+ const dimensionInfos: Dictionary<DataDimensionInfo> = {};
+ const dimensionNames = [];
+ const invertedIndicesMap: Dictionary<number[]> = {};
+
+ for (let i = 0; i < dimensions.length; i++) {
+ // Use the original dimensions[i], where other flag props may exists.
+ const dimInfoInput = dimensions[i];
+
+ const dimensionInfo: DataDimensionInfo =
+ zrUtil.isString(dimInfoInput)
+ ? new DataDimensionInfo({name: dimInfoInput})
+ : !(dimInfoInput instanceof DataDimensionInfo)
+ ? new DataDimensionInfo(dimInfoInput)
+ : dimInfoInput;
+
+ const dimensionName = dimensionInfo.name;
+ dimensionInfo.type = dimensionInfo.type || 'float';
+ if (!dimensionInfo.coordDim) {
+ dimensionInfo.coordDim = dimensionName;
+ dimensionInfo.coordDimIndex = 0;
+ }
+
+ const otherDims = dimensionInfo.otherDims = dimensionInfo.otherDims || {};
+ dimensionNames.push(dimensionName);
+ dimensionInfos[dimensionName] = dimensionInfo;
+
+ dimensionInfo.index = i;
+
+ if (dimensionInfo.createInvertedIndices) {
+ invertedIndicesMap[dimensionName] = [];
+ }
+ if (otherDims.itemName === 0) {
+ this._nameDimIdx = i;
+ }
+ if (otherDims.itemId === 0) {
+ this._idDimIdx = i;
+ }
+ }
+
+ this.dimensions = dimensionNames;
+ this._dimensionInfos = dimensionInfos;
+ this.hostModel = hostModel;
+
+ // Cache summary info for fast visit. See "dimensionHelper".
+ this._dimensionsSummary = summarizeDimensions(this);
+
+ this._invertedIndicesMap = invertedIndicesMap;
+
+ this.userOutput = this._dimensionsSummary.userOutput;
+ }
+
+ /**
+ * The meanings of the input parameter `dim`:
+ *
+ * + If dim is a number (e.g., `1`), it means the index of the dimension.
+ * For example, `getDimension(0)` will return 'x' or 'lng' or 'radius'.
+ * + If dim is a number-like string (e.g., `"1"`):
+ * + If there is the same concrete dim name defined in `this.dimensions`, it means that concrete name.
+ * + If not, it will be converted to a number, which means the index of the dimension.
+ * (why? because of the backward compatbility. We have been tolerating number-like string in
+ * dimension setting, although now it seems that it is not a good idea.)
+ * For example, `visualMap[i].dimension: "1"` is the same meaning as `visualMap[i].dimension: 1`,
+ * if no dimension name is defined as `"1"`.
+ * + If dim is a not-number-like string, it means the concrete dim name.
+ * For example, it can be be default name `"x"`, `"y"`, `"z"`, `"lng"`, `"lat"`, `"angle"`, `"radius"`,
+ * or customized in `dimensions` property of option like `"age"`.
+ *
+ * Get dimension name
+ * @param dim See above.
+ * @return Concrete dim name.
+ */
+ getDimension(dim: DimensionLoose): DimensionName {
+ if (typeof dim === 'number'
+ // If being a number-like string but not being defined a dimension name.
+ || (!isNaN(dim as any) && !this._dimensionInfos.hasOwnProperty(dim))
+ ) {
+ dim = this.dimensions[dim as DimensionIndex];
+ }
+ return dim as DimensionName;
+ }
+
+ private _getStoreDimIndex(dim: DimensionLoose): DimensionIndex {
+ return this._store.getDimensionIndex(this.getDimension(dim));
+ }
+
+ /**
+ * Get type and calculation info of particular dimension
+ * @param dim
+ * Dimension can be concrete names like x, y, z, lng, lat, angle, radius
+ * Or a ordinal number. For example getDimensionInfo(0) will return 'x' or 'lng' or 'radius'
+ */
+ getDimensionInfo(dim: DimensionLoose): DataDimensionInfo {
+ // Do not clone, because there may be categories in dimInfo.
+ return this._dimensionInfos[this.getDimension(dim)];
+ }
+
+ /**
+ * concrete dimension name list on coord.
+ */
+ getDimensionsOnCoord(): DimensionName[] {
+ return this._dimensionsSummary.dataDimsOnCoord.slice();
+ }
+
+ /**
+ * @param coordDim
+ * @param idx A coordDim may map to more than one data dim.
+ * If not specified, return the first dim not extra.
+ * @return concrete data dim. If not found, return null/undefined
+ */
+ mapDimension(coordDim: DimensionName): DimensionName;
+ mapDimension(coordDim: DimensionName, idx: number): DimensionName;
+ mapDimension(coordDim: DimensionName, idx?: number): DimensionName {
+ const dimensionsSummary = this._dimensionsSummary;
+
+ if (idx == null) {
+ return dimensionsSummary.encodeFirstDimNotExtra[coordDim] as any;
+ }
+
+ const dims = dimensionsSummary.encode[coordDim];
+ return dims ? dims[idx as number] as any : null;
+ }
+
+ mapDimensionsAll(coordDim: DimensionName): DimensionName[] {
+ const dimensionsSummary = this._dimensionsSummary;
+ const dims = dimensionsSummary.encode[coordDim];
+ return (dims || []).slice();
+ }
+
+ getStorage() {
+ return this._store;
+ }
+
+ /**
+ * Initialize from data
+ * @param data source or data or data storage.
+ * @param nameList The name of a datum is used on data diff and
+ * default label/tooltip.
+ * A name can be specified in encode.itemName,
+ * or dataItem.name (only for series option data),
+ * or provided in nameList from outside.
+ */
+ initData(
+ data: Source | OptionSourceData | DataStorage | DataProvider,
+ nameList?: string[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ let store: DataStorage;
+ const dimensions = this.dimensions;
+ const dimensionInfos = map(dimensions, dimName => this._dimensionInfos[dimName]);
+ if (data instanceof DataStorage) {
+ if (data.syncDimensionTypes(dimensionInfos)) {
+ store = data;
+ }
+ // Sync failed
+ else {
+ // Needs to recreate data storage if it's not match the given dimension.
+ data = data.getSource();
+ }
+ }
+
+ if (!store) {
+ const provider = (isSourceInstance(data) || zrUtil.isArrayLike(data))
+ ? new DefaultDataProvider(data as Source | OptionSourceData, dimensions.length)
+ : data as DataProvider;
+ store = new DataStorage();
+ store.initData(provider, dimensionInfos, dimValueGetter);
+ }
+
+ this._store = store;
+
+ // Reset
+ this._nameList = (nameList || []).slice();
+ this._idList = [];
+ this._nameRepeatCount = {};
+
+ this._doInit(0, store.count());
+ }
+
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ */
+ appendData(data: ArrayLike<any>): void {
+ const range = this._store.appendData(data);
+ this._doInit(range[0], range[1]);
+ }
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ * This method does not modify `rawData` (`dataProvider`), but only
+ * add values to storage.
+ *
+ * The final count will be increased by `Math.max(values.length, names.length)`.
+ *
+ * @param values That is the SourceType: 'arrayRows', like
+ * [
+ * [12, 33, 44],
+ * [NaN, 43, 1],
+ * ['-', 'asdf', 0]
+ * ]
+ * Each item is exaclty cooresponding to a dimension.
+ */
+ appendValues(values: any[][], names?: string[]): void {
+ const {start, end} = this._store.appendValues(values, names.length);
+ const shouldMakeIdFromName = this._shouldMakeIdFromName();
+
+ this._updateOrdinalMeta();
+
+ if (names) {
+ for (let idx = start; idx < end; idx++) {
+ const sourceIdx = idx - start;
+ this._nameList[idx] = names[sourceIdx];
+ if (shouldMakeIdFromName) {
+ makeIdFromName(this, idx);
+ }
+ }
+ }
+ }
+
+ private _updateOrdinalMeta() {
+ const store = this._store;
+ const dimensions = this.dimensions;
+ for (let i = 0; i < dimensions.length; i++) {
+ const dimInfo = this._dimensionInfos[dimensions[i]];
+ if (dimInfo.ordinalMeta) {
+ const dimIdx = store.getDimensionIndex(dimensions[i]);
+ store.collectOrdinalMeta(dimIdx, dimInfo.ordinalMeta);
+ }
+ }
+ }
+
+ private _shouldMakeIdFromName() {
+ const provider = this._store.getProvider();
+ return this._idDimIdx == null
+ && provider.getSource().sourceFormat !== SOURCE_FORMAT_TYPED_ARRAY
+ && !provider.fillStorage;
+ }
+
+ private _doInit(start: number, end: number): void {
+ if (start >= end) {
+ return;
+ }
+
+ const store = this._store;
+ const provider = store.getProvider();
+
+ this._updateOrdinalMeta();
+
+ const nameList = this._nameList;
+ const idList = this._idList;
+ const sourceFormat = provider.getSource().sourceFormat;
+ const isFormatOriginal = sourceFormat === SOURCE_FORMAT_ORIGINAL;
+
+ // Each data item is value
+ // [1, 2]
+ // 2
+ // Bar chart, line chart which uses category axis
+ // only gives the 'y' value. 'x' value is the indices of category
+ // Use a tempValue to normalize the value to be a (x, y) value
+ // If dataItem is {name: ...} or {id: ...}, it has highest priority.
+ // This kind of ids and names are always stored `_nameList` and `_idList`.
+ if (isFormatOriginal && !provider.pure) {
+ const sharedDataItem = [] as OptionDataItem;
+ for (let idx = start; idx < end; idx++) {
+ // NOTICE: Try not to write things into dataItem
+ const dataItem = provider.getItem(idx, sharedDataItem);
+ if (!this.hasItemOption && isDataItemOption(dataItem)) {
+ this.hasItemOption = true;
+ }
+ if (dataItem) {
+ const itemName = (dataItem as any).name;
+ if (nameList[idx] == null && itemName != null) {
+ nameList[idx] = convertOptionIdName(itemName, null);
+ }
+ const itemId = (dataItem as any).id;
+ if (idList[idx] == null && itemId != null) {
+ idList[idx] = convertOptionIdName(itemId, null);
+ }
+ }
+ }
+ }
+
+ if (this._shouldMakeIdFromName()) {
+ for (let idx = start; idx < end; idx++) {
+ makeIdFromName(this, idx);
+ }
+ }
+
+ prepareInvertedIndex(this);
+ }
+ /**
+ * PENDING: In fact currently this function is only used to short-circuit
+ * the calling of `scale.unionExtentFromData` when data have been filtered by modules
+ * like "dataZoom". `scale.unionExtentFromData` is used to calculate data extent for series on
+ * an axis, but if a "axis related data filter module" is used, the extent of the axis have
+ * been fixed and no need to calling `scale.unionExtentFromData` actually.
+ * But if we add "custom data filter" in future, which is not "axis related", this method may
+ * be still needed.
+ *
+ * Optimize for the scenario that data is filtered by a given extent.
+ * Consider that if data amount is more than hundreds of thousand,
+ * extent calculation will cost more than 10ms and the cache will
+ * be erased because of the filtering.
+ */
+ getApproximateExtent(dim: DimensionLoose): [number, number] {
+ return this._approximateExtent[dim] || this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ /**
+ * Calculate extent on a filtered data might be time consuming.
+ * Approximate extent is only used for: calculte extent of filtered data outside.
+ */
+ setApproximateExtent(extent: [number, number], dim: DimensionLoose): void {
+ dim = this.getDimension(dim);
+ this._approximateExtent[dim] = extent.slice() as [number, number];
+ }
+
+ getCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY
+ ): DataCalculationInfo<HostModel>[CALC_INFO_KEY] {
+ return this._calculationInfo[key];
+ }
+
+ /**
+ * @param key or k-v object
+ */
+ setCalculationInfo(
+ key: DataCalculationInfo<HostModel>
+ ): void;
+ setCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY,
+ value: DataCalculationInfo<HostModel>[CALC_INFO_KEY]
+ ): void;
+ setCalculationInfo(
+ key: (keyof DataCalculationInfo<HostModel>) | DataCalculationInfo<HostModel>,
+ value?: DataCalculationInfo<HostModel>[keyof DataCalculationInfo<HostModel>]
+ ): void {
+ isObject(key)
+ ? zrUtil.extend(this._calculationInfo, key as object)
+ : ((this._calculationInfo as any)[key] = value);
+ }
+
+ /**
+ * @return Never be null/undefined. `number` will be converted to string. Becuase:
+ * In most cases, name is used in display, where returning a string is more convenient.
+ * In other cases, name is used in query (see `indexOfName`), where we can keep the
+ * rule that name `2` equals to name `'2'`.
+ */
+ getName(idx: number): string {
+ const rawIndex = this.getRawIndex(idx);
+ let name = this._nameList[rawIndex];
+ if (name == null && this._nameDimIdx != null) {
+ name = getIdNameFromStore(this, this._nameDimIdx, rawIndex);
+ }
+ if (name == null) {
+ name = '';
+ }
+ return name;
+ }
+
+ private _getCategory(dimIdx: number, idx: number) {
+ const ordinal = this._store.get(dimIdx, idx);
+ const ordinalMeta = this._store.getOrdinalMeta(dimIdx);
+ if (ordinalMeta) {
+ return ordinalMeta.categories[ordinal as OrdinalNumber];
+ }
+ return ordinal;
+ }
+
+ /**
+ * @return Never null/undefined. `number` will be converted to string. Becuase:
+ * In all cases having encountered at present, id is used in making diff comparison, which
+ * are usually based on hash map. We can keep the rule that the internal id are always string
+ * (treat `2` is the same as `'2'`) to make the related logic simple.
+ */
+ getId(idx: number): string {
+ return getId(this, this.getRawIndex(idx));
+ }
+
+ count() {
+ return this._store.count();
+ }
+ /**
+ * Get value. Return NaN if idx is out of range.
+ * @param dim Dim must be concrete name.
+ */
+ get(dim: DimensionName, idx: number): ParsedValue {
+ const store = this._store;
+ return store.get(store.getDimensionIndex(dim), idx);
+ }
+
+ getByRawIndex(dim: DimensionName, rawIdx: number): ParsedValue {
+ const store = this._store;
+ return store.getByRawIndex(store.getDimensionIndex(dim), rawIdx);
+ }
+
+ getIndices() {
+ return this._store.getIndices();
+ }
+
+ getDataExtent(dim: DimensionLoose): [number, number] {
+ return this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ getSum(dim: DimensionLoose): number {
+ return this._store.getSum(this._getStoreDimIndex(dim));
+ }
+
+ getMedian(dim: DimensionLoose): number {
+ return this._store.getMedian(this._getStoreDimIndex(dim));
+ }
+ /**
+ * Get value for multi dimensions.
+ * @param dimensions If ignored, using all dimensions.
+ */
+ getValues(idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[], idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[] | number, idx?: number): ParsedValue[] {
+ const store = this._store;
+ return zrUtil.isArray(dimensions)
+ ? store.getValues(map(dimensions, dim => this._getStoreDimIndex(dim)), idx)
+ : store.getValues(dimensions as number);
+ }
+
+ /**
+ * If value is NaN. Inlcuding '-'
+ * Only check the coord dimensions.
+ */
+ hasValue(idx: number): boolean {
+ const dataDimsOnCoord = this._dimensionsSummary.dataDimsOnCoord;
+ for (let i = 0, len = dataDimsOnCoord.length; i < len; i++) {
+ // Ordinal type originally can be string or number.
+ // But when an ordinal type is used on coord, it can
+ // not be string but only number. So we can also use isNaN.
+ if (isNaN(this.get(dataDimsOnCoord[i], idx) as any)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Retreive the index with given name
+ */
+ indexOfName(name: string): number {
+ for (let i = 0, len = this._store.count(); i < len; i++) {
+ if (this.getName(i) === name) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ getRawIndex(idx: number) {
+ return this._store.getRawIndex(idx);
+ }
+
+ indexOfRawIndex(rawIndex: number) {
+ return this._store.indexOfRawIndex(rawIndex);
+ }
+
+ /**
+ * Only support the dimension which inverted index created.
+ * Do not support other cases until required.
+ * @param dim concrete dim
+ * @param value ordinal index
+ * @return rawIndex
+ */
+ rawIndexOf(dim: DimensionName, value: OrdinalNumber): number {
+ const invertedIndices = dim && this._invertedIndicesMap[dim];
+ if (__DEV__) {
+ if (!invertedIndices) {
+ throw new Error('Do not supported yet');
+ }
+ }
+ const rawIndex = invertedIndices[value];
+ if (rawIndex == null || isNaN(rawIndex)) {
+ return INDEX_NOT_FOUND;
+ }
+ return rawIndex;
+ }
+
+ /**
+ * Retreive the index of nearest value
+ * @param dim
+ * @param value
+ * @param [maxDistance=Infinity]
+ * @return If and only if multiple indices has
+ * the same value, they are put to the result.
+ */
+ indicesOfNearest(dim: DimensionLoose, value: number, maxDistance?: number): number[] {
+ return this._store.indicesOfNearest(
+ this._getStoreDimIndex(dim),
+ value, maxDistance
+ );
+ }
+ /**
+ * Data iteration
+ * @param ctx default this
+ * @example
+ * list.each('x', function (x, idx) {});
+ * list.each(['x', 'y'], function (x, y, idx) {});
+ * list.each(function (idx) {})
+ */
+ each<Ctx>(cb: EachCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): void;
+ each<Ctx>(dims: DimensionLoose, cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose], cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: EachCb2<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: ItrParamDims, cb: EachCb<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(
+ dims: ItrParamDims | EachCb<Ctx>,
+ cb: EachCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): void {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store.each(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ }
+ /**
+ * Data filter
+ */
+ filterSelf<Ctx>(cb: FilterCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): this;
+ filterSelf<Ctx>(dims: DimensionLoose, cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose], cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: FilterCb2<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: ItrParamDims, cb: FilterCb<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(
+ dims: ItrParamDims | FilterCb<Ctx>,
+ cb: FilterCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ // Clone first
+ this._store = this._store.clone();
+ this._store.filterSelf(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+
+ return this;
+ }
+
+ /**
+ * Select data in range. (For optimization of filter)
+ * (Manually inline code, support 5 million data filtering in data zoom.)
+ */
+ selectRange(range: Record<string, [number, number]>): SeriesData {
+ 'use strict';
+
+ const innerRange: Record<number, [number, number]> = {};
+ const dims = zrUtil.keys(range);
+ const dimIndices: number[] = [];
+ zrUtil.each(dims, (dim) => {
+ const dimIdx = this._getStoreDimIndex(dim);
+ innerRange[dimIdx] = range[dim];
+ dimIndices.push(dimIdx);
+ });
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store = this._store.clone();
+ this._store.selectRange(innerRange);
+ return this;
+ }
+
+ /**
+ * Data mapping to a plain array
+ */
+ mapArray<Ctx, Cb extends MapArrayCb0<Ctx>>(cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-disable */
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: DimensionLoose, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: [DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb2<Ctx>>(dims: [DimensionLoose, DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb<Ctx>>(dims: ItrParamDims, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-enable */
+ mapArray<Ctx>(
+ dims: ItrParamDims | MapArrayCb<Ctx>,
+ cb: MapArrayCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): unknown[] {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ ctx = (ctx || this) as Ctx;
+
+ const result: unknown[] = [];
+ this.each(dims, function () {
+ result.push(cb && (cb as MapArrayCb<Ctx>).apply(this, arguments));
+ }, ctx);
+ return result;
+ }
+
+ /**
+ * Data mapping to a new List with given dimensions
+ */
+ map<Ctx>(dims: DimensionLoose, cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(dims: [DimensionLoose], cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ /* eslint-disable-next-line */
+ map<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: MapCb2<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(
+ dims: ItrParamDims,
+ cb: MapCb<Ctx>,
+ ctx?: Ctx,
+ ctxCompat?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || ctxCompat || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(
+ normalizeDimensions(dims), this._getStoreDimIndex, this
+ );
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ const list = cloneListForMapAndSample(this);
+ list._store = this._store.map(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ return list;
+ }
+
+ /**
+ * Large data down sampling on given dimension
+ * @param sampleIndex Sample index for name and id
+ */
+ downSample(
Review comment:
Yes I agree, `DataStorage` should be totally immutable. (The only exception is `appendData` and `appendValues`, which are restricted to only be called in `Series#getRawData`, which is the root of all of the clones).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r674490745
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
> Do we need to distinguish user defined string 'null' or 'undefined' from null / undefined here?
I don't understand what you mean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673344421
##########
File path: src/model/Series.ts
##########
@@ -385,14 +391,22 @@ class SeriesModel<Opt extends SeriesOption = SeriesOption> extends ComponentMode
inner(this).data = data;
}
- getSource(): Source {
- return inner(this).sourceManager.getSource();
+ getEncode() {
+ const encode = (this as Model<SeriesEncodeOptionMixin>).get('encode');
Review comment:
Need to
```ts
(this as Model<SeriesEncodeOptionMixin>).get('encode', true)
```
?
The original code is `.get('encode', true)` and there might be no scenario to set `encode` on option root and share with different series ?
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,79 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(endSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+
+ const source = this.getSource(sourceIndex);
+ const storeList = this._storeList;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this two props as key. Another fact `dimensions` will be checked when initializing SeriesData.
+ const sourceToInit = (endSource || source);
Review comment:
The `endSource` seems never be null/undefined, so here `source` will never be used ?
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,79 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
Review comment:
I think it might not necessary to share `source` instance between `sourceManager`s.
The `class Source` originally try to represents the option of `data`, `dimension`, ... of `series` and `dataset`, where `data`, `dimensions` probably be shared between different `source` instance, but `source` instance itself do not share between different `sourceManager`s.
I think it's seems not necessary to change the understanding above.
Otherwise it brings some code in Line 237, which are not easy to understand that why there is a different that line 237 only compare `seriesLayoutBy` and `sourceHeader` but line 379 also compare `dimensions`.
And it also brings code here, which is not easy to think through where the `source` instance from on earth (because of the call of `this._getUpstreamSourceManagers`)
The `dataStore` will be shard between `sourceManager`, it's seems enough in performance optimization?
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,79 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(endSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+
+ const source = this.getSource(sourceIndex);
+ const storeList = this._storeList;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this two props as key. Another fact `dimensions` will be checked when initializing SeriesData.
+ const sourceToInit = (endSource || source);
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(endSource, sourceReadKey);
+ }
+ else {
+ const dimensionsDefine = source.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (source && dimensionsDefine) {
Review comment:
Why not `endSource` ?
It seem not easy to understand what the `source` and `endSource` are here.
##########
File path: src/chart/helper/createSeriesDataFromArray.ts
##########
@@ -107,26 +99,92 @@ function createListFromArray(source: Source | OptionSourceData, seriesModel: Ser
if (!hasNameEncode && firstCategoryDimIndex != null) {
dimInfoList[firstCategoryDimIndex].otherDims.itemName = 0;
}
+ return firstCategoryDimIndex;
+}
- const stackCalculationInfo = enableDataStack(seriesModel, dimInfoList);
+function createListFromArray(
+ sourceOrStore: Source | OptionSourceData | DataStorage,
+ seriesModel: SeriesModel,
+ opt?: {
+ generateCoord?: string
+ useEncodeDefaulter?: boolean | EncodeDefaulter
+ // By default: auto. If `true`, create inverted indices for all ordinal dimension on coordSys.
+ createInvertedIndices?: boolean
+ }
+): SeriesData {
+ opt = opt || {};
- const list = new List(dimInfoList, seriesModel);
+ if (!isSourceInstance(sourceOrStore) && !isDataStorage(sourceOrStore)) {
+ sourceOrStore = createSourceFromSeriesDataOption(
+ sourceOrStore as OptionSourceData
+ );
+ }
- list.setCalculationInfo(stackCalculationInfo);
+ const source = isDataStorage(sourceOrStore) ? sourceOrStore.getSource() : sourceOrStore;
+ const coordSysInfo = getCoordSysInfoBySeries(seriesModel);
+ const coordSysDimDefs = getCoordSysDimDefs(seriesModel, coordSysInfo);
+ const useEncodeDefaulter = opt.useEncodeDefaulter;
- const dimValueGetter = (firstCategoryDimIndex != null && isNeedCompleteOrdinalData(source))
- ? function (this: List, itemOpt: any, dimName: string, dataIndex: number, dimIndex: number) {
- // Use dataIndex as ordinal value in categoryAxis
- return dimIndex === firstCategoryDimIndex
- ? dataIndex
- : this.defaultDimValueGetter(itemOpt, dimName, dataIndex, dimIndex);
- }
+ // Try to ignore unsed dimensions if sharing a high dimension datastorage
+ // 30 is an experience value.
+ const omitUnusedDimensions = isDataStorage(sourceOrStore) && sourceOrStore.getDimensionCount() > 30;
+ const encodeDefaulter = zrUtil.isFunction(useEncodeDefaulter)
+ ? useEncodeDefaulter
+ : useEncodeDefaulter
+ ? zrUtil.curry(makeSeriesEncodeForAxisCoordSys, coordSysDimDefs, seriesModel)
: null;
- list.hasItemOption = false;
- list.initData(source, null, dimValueGetter);
+ const createDimensionOptions = {
+ coordDimensions: coordSysDimDefs,
+ generateCoord: opt.generateCoord,
+ encodeDefine: seriesModel.getEncode()
+ // NOTE: If we call createDimensions on same source multiple times.
+ // It will break the encodeDefaulter which has sideeffects.
+ // So we prepare the default encode here instead of passing encoderDefaulter function.
+ || (encodeDefaulter && encodeDefaulter(
+ source, getDimCount(source, coordSysDimDefs, source.dimensionsDefine || [])
+ )),
+ omitUnusedDimensions
+ };
+ let dimInfoList = createDimensions(sourceOrStore, createDimensionOptions);
+ let firstCategoryDimIndex = injectOrdinalMeta(dimInfoList, opt.createInvertedIndices, coordSysInfo);
+
+ if (omitUnusedDimensions) {
+ // sourceOrStore
+ if (!(sourceOrStore as DataStorage).syncDimensionTypes(dimInfoList)) {
+ dimInfoList = createDimensions(sourceOrStore, zrUtil.extend(createDimensionOptions, {
+ omitUnusedDimensions: true
+ }));
+ // Fallback
+ firstCategoryDimIndex = injectOrdinalMeta(
+ dimInfoList, opt.createInvertedIndices, coordSysInfo
+ );
+ sourceOrStore = source;
+ }
+ }
Review comment:
Not sure why call them twice rather than
```ts
if (omitUnusedDimensions && !(sourceOrStore as DataStorage).syncDimensionTypes(dimInfoList)) {
dimInfoList = createDimensions(...);
firstCategoryDimIndex = injectOrdinalMeta(...);
sourceOrStore = source;
}
else {
dimInfoList = createDimensions(...);
firstCategoryDimIndex = injectOrdinalMeta(...);
}
```
##########
File path: src/chart/helper/createSeriesDataFromArray.ts
##########
@@ -107,26 +99,92 @@ function createListFromArray(source: Source | OptionSourceData, seriesModel: Ser
if (!hasNameEncode && firstCategoryDimIndex != null) {
dimInfoList[firstCategoryDimIndex].otherDims.itemName = 0;
}
+ return firstCategoryDimIndex;
+}
- const stackCalculationInfo = enableDataStack(seriesModel, dimInfoList);
+function createListFromArray(
+ sourceOrStore: Source | OptionSourceData | DataStorage,
+ seriesModel: SeriesModel,
+ opt?: {
+ generateCoord?: string
+ useEncodeDefaulter?: boolean | EncodeDefaulter
+ // By default: auto. If `true`, create inverted indices for all ordinal dimension on coordSys.
+ createInvertedIndices?: boolean
+ }
+): SeriesData {
+ opt = opt || {};
- const list = new List(dimInfoList, seriesModel);
+ if (!isSourceInstance(sourceOrStore) && !isDataStorage(sourceOrStore)) {
+ sourceOrStore = createSourceFromSeriesDataOption(
+ sourceOrStore as OptionSourceData
+ );
+ }
- list.setCalculationInfo(stackCalculationInfo);
+ const source = isDataStorage(sourceOrStore) ? sourceOrStore.getSource() : sourceOrStore;
+ const coordSysInfo = getCoordSysInfoBySeries(seriesModel);
+ const coordSysDimDefs = getCoordSysDimDefs(seriesModel, coordSysInfo);
+ const useEncodeDefaulter = opt.useEncodeDefaulter;
- const dimValueGetter = (firstCategoryDimIndex != null && isNeedCompleteOrdinalData(source))
- ? function (this: List, itemOpt: any, dimName: string, dataIndex: number, dimIndex: number) {
- // Use dataIndex as ordinal value in categoryAxis
- return dimIndex === firstCategoryDimIndex
- ? dataIndex
- : this.defaultDimValueGetter(itemOpt, dimName, dataIndex, dimIndex);
- }
+ // Try to ignore unsed dimensions if sharing a high dimension datastorage
+ // 30 is an experience value.
+ const omitUnusedDimensions = isDataStorage(sourceOrStore) && sourceOrStore.getDimensionCount() > 30;
+ const encodeDefaulter = zrUtil.isFunction(useEncodeDefaulter)
+ ? useEncodeDefaulter
+ : useEncodeDefaulter
+ ? zrUtil.curry(makeSeriesEncodeForAxisCoordSys, coordSysDimDefs, seriesModel)
: null;
- list.hasItemOption = false;
- list.initData(source, null, dimValueGetter);
+ const createDimensionOptions = {
+ coordDimensions: coordSysDimDefs,
+ generateCoord: opt.generateCoord,
+ encodeDefine: seriesModel.getEncode()
+ // NOTE: If we call createDimensions on same source multiple times.
+ // It will break the encodeDefaulter which has sideeffects.
Review comment:
Yes I've understood that the side-effect is indeed annoying.
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `sysDims`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimsDef` and `opt.encodeDef`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
Review comment:
In comment above:
`sysDims` =renamed=> `opt.coordDimensions`.
`opt.dimsDef` =renamed=> `opt.dimensionsDefine`
`opt.encodeDefine` =renamed=> `opt.encodeDefine`
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `sysDims`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimsDef` and `opt.encodeDef`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
Review comment:
It depends on that `source` instance is immutable. Do we need to add a comment on `class Source` to caution that keep it immutable in future?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673627356
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `sysDims`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimsDef` and `opt.encodeDef`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r674499977
##########
File path: src/data/SeriesData.ts
##########
@@ -0,0 +1,1338 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+/**
+ * List for data storage
+ */
+
+import * as zrUtil from 'zrender/src/core/util';
+import {PathStyleProps} from 'zrender/src/graphic/Path';
+import Model from '../model/Model';
+import DataDiffer from './DataDiffer';
+import {DataProvider, DefaultDataProvider} from './helper/dataProvider';
+import {summarizeDimensions, DimensionSummary} from './helper/dimensionHelper';
+import DataDimensionInfo from './DataDimensionInfo';
+import {ArrayLike, Dictionary, FunctionPropertyNames} from 'zrender/src/core/types';
+import Element from 'zrender/src/Element';
+import {
+ DimensionIndex, DimensionName, DimensionLoose, OptionDataItem,
+ ParsedValue, ParsedValueNumeric, DimensionUserOuput,
+ ModelOption, SeriesDataType, OptionSourceData, SOURCE_FORMAT_TYPED_ARRAY, SOURCE_FORMAT_ORIGINAL,
+ DecalObject,
+ OrdinalNumber
+} from '../util/types';
+import {convertOptionIdName, isDataItemOption} from '../util/model';
+import { setCommonECData } from '../util/innerStore';
+import type Graph from './Graph';
+import type Tree from './Tree';
+import type { VisualMeta } from '../component/visualMap/VisualMapModel';
+import {isSourceInstance, Source} from './Source';
+import { LineStyleProps } from '../model/mixin/lineStyle';
+import DataStorage, { DimValueGetter } from './DataStorage';
+
+const isObject = zrUtil.isObject;
+const map = zrUtil.map;
+
+const CtorInt32Array = typeof Int32Array === 'undefined' ? Array : Int32Array;
+
+// Use prefix to avoid index to be the same as otherIdList[idx],
+// which will cause weird udpate animation.
+const ID_PREFIX = 'e\0\0';
+
+const INDEX_NOT_FOUND = -1;
+
+type NameRepeatCount = {[name: string]: number};
+type ItrParamDims = DimensionLoose | Array<DimensionLoose>;
+// If Ctx not specified, use List as Ctx
+type CtxOrList<Ctx> = unknown extends Ctx ? SeriesData : Ctx;
+type EachCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => void;
+type EachCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => void;
+type EachCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => void;
+type FilterCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => boolean;
+type FilterCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => boolean;
+type FilterCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => boolean;
+type FilterCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => boolean;
+type MapArrayCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => any;
+type MapArrayCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => any;
+type MapArrayCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => any;
+type MapArrayCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => any;
+type MapCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => ParsedValue | ParsedValue[];
+type MapCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) =>
+ ParsedValue | ParsedValue[];
+type MapCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => ParsedValue | ParsedValue[];
+
+const TRANSFERABLE_PROPERTIES = [
+ 'hasItemOption', '_nameList', '_idList', '_invertedIndicesMap',
+ '_rawData', '_dimValueGetter',
+ '_nameDimIdx', '_idDimIdx', '_nameRepeatCount'
+];
+const CLONE_PROPERTIES = [
+ '_approximateExtent'
+];
+
+export interface DefaultDataVisual {
+ style: PathStyleProps
+ // Draw type determined which prop should be set with encoded color.
+ // It's only available on the global visual. Use getVisual('drawType') to access it.
+ // It will be set in visual/style.ts module in the first priority.
+ drawType: 'fill' | 'stroke'
+
+ symbol?: string
+ symbolSize?: number | number[]
+ symbolRotate?: number
+ symbolKeepAspect?: boolean
+ symbolOffset?: string | number | (string | number)[]
+
+ liftZ?: number
+ // For legend.
+ legendIcon?: string
+ legendLineStyle?: LineStyleProps
+
+ // visualMap will inject visualMeta data
+ visualMeta?: VisualMeta[]
+
+ // If color is encoded from palette
+ colorFromPalette?: boolean
+
+ decal?: DecalObject
+}
+
+export interface DataCalculationInfo<SERIES_MODEL> {
+ stackedDimension: string;
+ stackedByDimension: string;
+ isStackedByIndex: boolean;
+ stackedOverDimension: string;
+ stackResultDimension: string;
+ stackedOnSeries?: SERIES_MODEL;
+}
+
+// -----------------------------
+// Internal method declarations:
+// -----------------------------
+let prepareInvertedIndex: (data: SeriesData) => void;
+let getId: (data: SeriesData, rawIndex: number) => string;
+let getIdNameFromStore: (data: SeriesData, dimIdx: number, dataIdx: number) => string;
+let normalizeDimensions: (dimensions: ItrParamDims) => Array<DimensionLoose>;
+let validateDimensions: (data: SeriesData, dims: DimensionIndex[]) => void;
+let transferProperties: (target: SeriesData, source: SeriesData) => void;
+let cloneListForMapAndSample: (original: SeriesData) => SeriesData;
+let makeIdFromName: (data: SeriesData, idx: number) => void;
+
+class SeriesData<
+ HostModel extends Model = Model,
+ Visual extends DefaultDataVisual = DefaultDataVisual
+> {
+
+ readonly type = 'list';
+
+ readonly dimensions: string[];
+
+ // Infomation of each data dimension, like data type.
+ private _dimensionInfos: {[dimName: string]: DataDimensionInfo};
+
+ readonly hostModel: HostModel;
+
+ /**
+ * @readonly
+ */
+ dataType: SeriesDataType;
+
+ /**
+ * @readonly
+ * Host graph if List is used to store graph nodes / edges.
+ */
+ graph?: Graph;
+
+ /**
+ * @readonly
+ * Host tree if List is used to store tree ndoes.
+ */
+ tree?: Tree;
+
+ private _store: DataStorage;
+
+ private _nameList: string[] = [];
+ private _idList: string[] = [];
+
+ // Models of data option is stored sparse for optimizing memory cost
+ // Never used yet (not used yet).
+ // private _optionModels: Model[] = [];
+
+ // Global visual properties after visual coding
+ private _visual: Dictionary<any> = {};
+
+ // Globel layout properties.
+ private _layout: Dictionary<any> = {};
+
+ // Item visual properties after visual coding
+ private _itemVisuals: Dictionary<any>[] = [];
+
+ // Item layout properties after layout
+ private _itemLayouts: any[] = [];
+
+ // Graphic elemnents
+ private _graphicEls: Element[] = [];
+
+ // key: dim, value: extent
+ private _approximateExtent: {[dimName: string]: [number, number]} = {};
+
+ private _dimensionsSummary: DimensionSummary;
+
+ private _invertedIndicesMap: {[dimName: string]: ArrayLike<number>};
+
+ private _calculationInfo: DataCalculationInfo<HostModel> = {} as DataCalculationInfo<HostModel>;
+
+ // User output info of this data.
+ // DO NOT use it in other places!
+ // When preparing user params for user callbacks, we have
+ // to clone these inner data structures to prevent users
+ // from modifying them to effect built-in logic. And for
+ // performance consideration we make this `userOutput` to
+ // avoid clone them too many times.
+ readonly userOutput: DimensionUserOuput;
+
+ // Having detected that there is data item is non primitive type
+ // (in type `OptionDataItemObject`).
+ // Like `data: [ { value: xx, itemStyle: {...} }, ...]`
+ // At present it only happen in `SOURCE_FORMAT_ORIGINAL`.
+ hasItemOption: boolean = false;
+
+ // id or name is used on dynamic data, mapping old and new items.
+ // When generating id from name, avoid repeat.
+ private _nameRepeatCount: NameRepeatCount;
+ private _nameDimIdx: number;
+ private _idDimIdx: number;
+
+ private __wrappedMethods: string[];
+
+ // Methods that create a new list based on this list should be listed here.
+ // Notice that those method should `RETURN` the new list.
+ TRANSFERABLE_METHODS = ['cloneShallow', 'downSample', 'lttbDownSample', 'map'] as const;
+ // Methods that change indices of this list should be listed here.
+ CHANGABLE_METHODS = ['filterSelf', 'selectRange'] as const;
+ DOWNSAMPLE_METHODS = ['downSample', 'lttbDownSample'] as const;
+
+ /**
+ * @param dimensions
+ * For example, ['someDimName', {name: 'someDimName', type: 'someDimType'}, ...].
+ * Dimensions should be concrete names like x, y, z, lng, lat, angle, radius
+ */
+ constructor(dimensions: Array<string | object | DataDimensionInfo>, hostModel: HostModel) {
+ dimensions = dimensions || ['x', 'y'];
+
+ const dimensionInfos: Dictionary<DataDimensionInfo> = {};
+ const dimensionNames = [];
+ const invertedIndicesMap: Dictionary<number[]> = {};
+
+ for (let i = 0; i < dimensions.length; i++) {
+ // Use the original dimensions[i], where other flag props may exists.
+ const dimInfoInput = dimensions[i];
+
+ const dimensionInfo: DataDimensionInfo =
+ zrUtil.isString(dimInfoInput)
+ ? new DataDimensionInfo({name: dimInfoInput})
+ : !(dimInfoInput instanceof DataDimensionInfo)
+ ? new DataDimensionInfo(dimInfoInput)
+ : dimInfoInput;
+
+ const dimensionName = dimensionInfo.name;
+ dimensionInfo.type = dimensionInfo.type || 'float';
+ if (!dimensionInfo.coordDim) {
+ dimensionInfo.coordDim = dimensionName;
+ dimensionInfo.coordDimIndex = 0;
+ }
+
+ const otherDims = dimensionInfo.otherDims = dimensionInfo.otherDims || {};
+ dimensionNames.push(dimensionName);
+ dimensionInfos[dimensionName] = dimensionInfo;
+
+ dimensionInfo.index = i;
+
+ if (dimensionInfo.createInvertedIndices) {
+ invertedIndicesMap[dimensionName] = [];
+ }
+ if (otherDims.itemName === 0) {
+ this._nameDimIdx = i;
+ }
+ if (otherDims.itemId === 0) {
+ this._idDimIdx = i;
+ }
+ }
+
+ this.dimensions = dimensionNames;
+ this._dimensionInfos = dimensionInfos;
+ this.hostModel = hostModel;
+
+ // Cache summary info for fast visit. See "dimensionHelper".
+ this._dimensionsSummary = summarizeDimensions(this);
+
+ this._invertedIndicesMap = invertedIndicesMap;
+
+ this.userOutput = this._dimensionsSummary.userOutput;
+ }
+
+ /**
+ * The meanings of the input parameter `dim`:
+ *
+ * + If dim is a number (e.g., `1`), it means the index of the dimension.
+ * For example, `getDimension(0)` will return 'x' or 'lng' or 'radius'.
+ * + If dim is a number-like string (e.g., `"1"`):
+ * + If there is the same concrete dim name defined in `this.dimensions`, it means that concrete name.
+ * + If not, it will be converted to a number, which means the index of the dimension.
+ * (why? because of the backward compatbility. We have been tolerating number-like string in
+ * dimension setting, although now it seems that it is not a good idea.)
+ * For example, `visualMap[i].dimension: "1"` is the same meaning as `visualMap[i].dimension: 1`,
+ * if no dimension name is defined as `"1"`.
+ * + If dim is a not-number-like string, it means the concrete dim name.
+ * For example, it can be be default name `"x"`, `"y"`, `"z"`, `"lng"`, `"lat"`, `"angle"`, `"radius"`,
+ * or customized in `dimensions` property of option like `"age"`.
+ *
+ * Get dimension name
+ * @param dim See above.
+ * @return Concrete dim name.
+ */
+ getDimension(dim: DimensionLoose): DimensionName {
+ if (typeof dim === 'number'
+ // If being a number-like string but not being defined a dimension name.
+ || (!isNaN(dim as any) && !this._dimensionInfos.hasOwnProperty(dim))
+ ) {
+ dim = this.dimensions[dim as DimensionIndex];
+ }
+ return dim as DimensionName;
+ }
+
+ private _getStoreDimIndex(dim: DimensionLoose): DimensionIndex {
+ return this._store.getDimensionIndex(this.getDimension(dim));
+ }
+
+ /**
+ * Get type and calculation info of particular dimension
+ * @param dim
+ * Dimension can be concrete names like x, y, z, lng, lat, angle, radius
+ * Or a ordinal number. For example getDimensionInfo(0) will return 'x' or 'lng' or 'radius'
+ */
+ getDimensionInfo(dim: DimensionLoose): DataDimensionInfo {
+ // Do not clone, because there may be categories in dimInfo.
+ return this._dimensionInfos[this.getDimension(dim)];
+ }
+
+ /**
+ * concrete dimension name list on coord.
+ */
+ getDimensionsOnCoord(): DimensionName[] {
+ return this._dimensionsSummary.dataDimsOnCoord.slice();
+ }
+
+ /**
+ * @param coordDim
+ * @param idx A coordDim may map to more than one data dim.
+ * If not specified, return the first dim not extra.
+ * @return concrete data dim. If not found, return null/undefined
+ */
+ mapDimension(coordDim: DimensionName): DimensionName;
+ mapDimension(coordDim: DimensionName, idx: number): DimensionName;
+ mapDimension(coordDim: DimensionName, idx?: number): DimensionName {
+ const dimensionsSummary = this._dimensionsSummary;
+
+ if (idx == null) {
+ return dimensionsSummary.encodeFirstDimNotExtra[coordDim] as any;
+ }
+
+ const dims = dimensionsSummary.encode[coordDim];
+ return dims ? dims[idx as number] as any : null;
+ }
+
+ mapDimensionsAll(coordDim: DimensionName): DimensionName[] {
+ const dimensionsSummary = this._dimensionsSummary;
+ const dims = dimensionsSummary.encode[coordDim];
+ return (dims || []).slice();
+ }
+
+ getStorage() {
+ return this._store;
+ }
+
+ /**
+ * Initialize from data
+ * @param data source or data or data storage.
+ * @param nameList The name of a datum is used on data diff and
+ * default label/tooltip.
+ * A name can be specified in encode.itemName,
+ * or dataItem.name (only for series option data),
+ * or provided in nameList from outside.
+ */
+ initData(
+ data: Source | OptionSourceData | DataStorage | DataProvider,
+ nameList?: string[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ let store: DataStorage;
+ const dimensions = this.dimensions;
+ const dimensionInfos = map(dimensions, dimName => this._dimensionInfos[dimName]);
+ if (data instanceof DataStorage) {
+ if (data.syncDimensionTypes(dimensionInfos)) {
+ store = data;
+ }
+ // Sync failed
+ else {
+ // Needs to recreate data storage if it's not match the given dimension.
+ data = data.getSource();
+ }
+ }
+
+ if (!store) {
+ const provider = (isSourceInstance(data) || zrUtil.isArrayLike(data))
+ ? new DefaultDataProvider(data as Source | OptionSourceData, dimensions.length)
+ : data as DataProvider;
+ store = new DataStorage();
+ store.initData(provider, dimensionInfos, dimValueGetter);
+ }
+
+ this._store = store;
+
+ // Reset
+ this._nameList = (nameList || []).slice();
+ this._idList = [];
+ this._nameRepeatCount = {};
+
+ this._doInit(0, store.count());
+ }
+
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ */
+ appendData(data: ArrayLike<any>): void {
+ const range = this._store.appendData(data);
+ this._doInit(range[0], range[1]);
+ }
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ * This method does not modify `rawData` (`dataProvider`), but only
+ * add values to storage.
+ *
+ * The final count will be increased by `Math.max(values.length, names.length)`.
+ *
+ * @param values That is the SourceType: 'arrayRows', like
+ * [
+ * [12, 33, 44],
+ * [NaN, 43, 1],
+ * ['-', 'asdf', 0]
+ * ]
+ * Each item is exaclty cooresponding to a dimension.
+ */
+ appendValues(values: any[][], names?: string[]): void {
+ const {start, end} = this._store.appendValues(values, names.length);
+ const shouldMakeIdFromName = this._shouldMakeIdFromName();
+
+ this._updateOrdinalMeta();
+
+ if (names) {
+ for (let idx = start; idx < end; idx++) {
+ const sourceIdx = idx - start;
+ this._nameList[idx] = names[sourceIdx];
+ if (shouldMakeIdFromName) {
+ makeIdFromName(this, idx);
+ }
+ }
+ }
+ }
+
+ private _updateOrdinalMeta() {
+ const store = this._store;
+ const dimensions = this.dimensions;
+ for (let i = 0; i < dimensions.length; i++) {
+ const dimInfo = this._dimensionInfos[dimensions[i]];
+ if (dimInfo.ordinalMeta) {
+ const dimIdx = store.getDimensionIndex(dimensions[i]);
+ store.collectOrdinalMeta(dimIdx, dimInfo.ordinalMeta);
+ }
+ }
+ }
+
+ private _shouldMakeIdFromName() {
+ const provider = this._store.getProvider();
+ return this._idDimIdx == null
+ && provider.getSource().sourceFormat !== SOURCE_FORMAT_TYPED_ARRAY
+ && !provider.fillStorage;
+ }
+
+ private _doInit(start: number, end: number): void {
+ if (start >= end) {
+ return;
+ }
+
+ const store = this._store;
+ const provider = store.getProvider();
+
+ this._updateOrdinalMeta();
+
+ const nameList = this._nameList;
+ const idList = this._idList;
+ const sourceFormat = provider.getSource().sourceFormat;
+ const isFormatOriginal = sourceFormat === SOURCE_FORMAT_ORIGINAL;
+
+ // Each data item is value
+ // [1, 2]
+ // 2
+ // Bar chart, line chart which uses category axis
+ // only gives the 'y' value. 'x' value is the indices of category
+ // Use a tempValue to normalize the value to be a (x, y) value
+ // If dataItem is {name: ...} or {id: ...}, it has highest priority.
+ // This kind of ids and names are always stored `_nameList` and `_idList`.
+ if (isFormatOriginal && !provider.pure) {
+ const sharedDataItem = [] as OptionDataItem;
+ for (let idx = start; idx < end; idx++) {
+ // NOTICE: Try not to write things into dataItem
+ const dataItem = provider.getItem(idx, sharedDataItem);
+ if (!this.hasItemOption && isDataItemOption(dataItem)) {
+ this.hasItemOption = true;
+ }
+ if (dataItem) {
+ const itemName = (dataItem as any).name;
+ if (nameList[idx] == null && itemName != null) {
+ nameList[idx] = convertOptionIdName(itemName, null);
+ }
+ const itemId = (dataItem as any).id;
+ if (idList[idx] == null && itemId != null) {
+ idList[idx] = convertOptionIdName(itemId, null);
+ }
+ }
+ }
+ }
+
+ if (this._shouldMakeIdFromName()) {
+ for (let idx = start; idx < end; idx++) {
+ makeIdFromName(this, idx);
+ }
+ }
+
+ prepareInvertedIndex(this);
+ }
+ /**
+ * PENDING: In fact currently this function is only used to short-circuit
+ * the calling of `scale.unionExtentFromData` when data have been filtered by modules
+ * like "dataZoom". `scale.unionExtentFromData` is used to calculate data extent for series on
+ * an axis, but if a "axis related data filter module" is used, the extent of the axis have
+ * been fixed and no need to calling `scale.unionExtentFromData` actually.
+ * But if we add "custom data filter" in future, which is not "axis related", this method may
+ * be still needed.
+ *
+ * Optimize for the scenario that data is filtered by a given extent.
+ * Consider that if data amount is more than hundreds of thousand,
+ * extent calculation will cost more than 10ms and the cache will
+ * be erased because of the filtering.
+ */
+ getApproximateExtent(dim: DimensionLoose): [number, number] {
+ return this._approximateExtent[dim] || this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ /**
+ * Calculate extent on a filtered data might be time consuming.
+ * Approximate extent is only used for: calculte extent of filtered data outside.
+ */
+ setApproximateExtent(extent: [number, number], dim: DimensionLoose): void {
+ dim = this.getDimension(dim);
+ this._approximateExtent[dim] = extent.slice() as [number, number];
+ }
+
+ getCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY
+ ): DataCalculationInfo<HostModel>[CALC_INFO_KEY] {
+ return this._calculationInfo[key];
+ }
+
+ /**
+ * @param key or k-v object
+ */
+ setCalculationInfo(
+ key: DataCalculationInfo<HostModel>
+ ): void;
+ setCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY,
+ value: DataCalculationInfo<HostModel>[CALC_INFO_KEY]
+ ): void;
+ setCalculationInfo(
+ key: (keyof DataCalculationInfo<HostModel>) | DataCalculationInfo<HostModel>,
+ value?: DataCalculationInfo<HostModel>[keyof DataCalculationInfo<HostModel>]
+ ): void {
+ isObject(key)
+ ? zrUtil.extend(this._calculationInfo, key as object)
+ : ((this._calculationInfo as any)[key] = value);
+ }
+
+ /**
+ * @return Never be null/undefined. `number` will be converted to string. Becuase:
+ * In most cases, name is used in display, where returning a string is more convenient.
+ * In other cases, name is used in query (see `indexOfName`), where we can keep the
+ * rule that name `2` equals to name `'2'`.
+ */
+ getName(idx: number): string {
+ const rawIndex = this.getRawIndex(idx);
+ let name = this._nameList[rawIndex];
+ if (name == null && this._nameDimIdx != null) {
+ name = getIdNameFromStore(this, this._nameDimIdx, rawIndex);
+ }
+ if (name == null) {
+ name = '';
+ }
+ return name;
+ }
+
+ private _getCategory(dimIdx: number, idx: number) {
+ const ordinal = this._store.get(dimIdx, idx);
+ const ordinalMeta = this._store.getOrdinalMeta(dimIdx);
+ if (ordinalMeta) {
+ return ordinalMeta.categories[ordinal as OrdinalNumber];
+ }
+ return ordinal;
+ }
+
+ /**
+ * @return Never null/undefined. `number` will be converted to string. Becuase:
+ * In all cases having encountered at present, id is used in making diff comparison, which
+ * are usually based on hash map. We can keep the rule that the internal id are always string
+ * (treat `2` is the same as `'2'`) to make the related logic simple.
+ */
+ getId(idx: number): string {
+ return getId(this, this.getRawIndex(idx));
+ }
+
+ count() {
+ return this._store.count();
+ }
+ /**
+ * Get value. Return NaN if idx is out of range.
+ * @param dim Dim must be concrete name.
+ */
+ get(dim: DimensionName, idx: number): ParsedValue {
+ const store = this._store;
+ return store.get(store.getDimensionIndex(dim), idx);
+ }
+
+ getByRawIndex(dim: DimensionName, rawIdx: number): ParsedValue {
+ const store = this._store;
+ return store.getByRawIndex(store.getDimensionIndex(dim), rawIdx);
+ }
+
+ getIndices() {
+ return this._store.getIndices();
+ }
+
+ getDataExtent(dim: DimensionLoose): [number, number] {
+ return this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ getSum(dim: DimensionLoose): number {
+ return this._store.getSum(this._getStoreDimIndex(dim));
+ }
+
+ getMedian(dim: DimensionLoose): number {
+ return this._store.getMedian(this._getStoreDimIndex(dim));
+ }
+ /**
+ * Get value for multi dimensions.
+ * @param dimensions If ignored, using all dimensions.
+ */
+ getValues(idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[], idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[] | number, idx?: number): ParsedValue[] {
+ const store = this._store;
+ return zrUtil.isArray(dimensions)
+ ? store.getValues(map(dimensions, dim => this._getStoreDimIndex(dim)), idx)
+ : store.getValues(dimensions as number);
+ }
+
+ /**
+ * If value is NaN. Inlcuding '-'
+ * Only check the coord dimensions.
+ */
+ hasValue(idx: number): boolean {
+ const dataDimsOnCoord = this._dimensionsSummary.dataDimsOnCoord;
+ for (let i = 0, len = dataDimsOnCoord.length; i < len; i++) {
+ // Ordinal type originally can be string or number.
+ // But when an ordinal type is used on coord, it can
+ // not be string but only number. So we can also use isNaN.
+ if (isNaN(this.get(dataDimsOnCoord[i], idx) as any)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Retreive the index with given name
+ */
+ indexOfName(name: string): number {
+ for (let i = 0, len = this._store.count(); i < len; i++) {
+ if (this.getName(i) === name) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ getRawIndex(idx: number) {
+ return this._store.getRawIndex(idx);
+ }
+
+ indexOfRawIndex(rawIndex: number) {
+ return this._store.indexOfRawIndex(rawIndex);
+ }
+
+ /**
+ * Only support the dimension which inverted index created.
+ * Do not support other cases until required.
+ * @param dim concrete dim
+ * @param value ordinal index
+ * @return rawIndex
+ */
+ rawIndexOf(dim: DimensionName, value: OrdinalNumber): number {
+ const invertedIndices = dim && this._invertedIndicesMap[dim];
+ if (__DEV__) {
+ if (!invertedIndices) {
+ throw new Error('Do not supported yet');
+ }
+ }
+ const rawIndex = invertedIndices[value];
+ if (rawIndex == null || isNaN(rawIndex)) {
+ return INDEX_NOT_FOUND;
+ }
+ return rawIndex;
+ }
+
+ /**
+ * Retreive the index of nearest value
+ * @param dim
+ * @param value
+ * @param [maxDistance=Infinity]
+ * @return If and only if multiple indices has
+ * the same value, they are put to the result.
+ */
+ indicesOfNearest(dim: DimensionLoose, value: number, maxDistance?: number): number[] {
+ return this._store.indicesOfNearest(
+ this._getStoreDimIndex(dim),
+ value, maxDistance
+ );
+ }
+ /**
+ * Data iteration
+ * @param ctx default this
+ * @example
+ * list.each('x', function (x, idx) {});
+ * list.each(['x', 'y'], function (x, y, idx) {});
+ * list.each(function (idx) {})
+ */
+ each<Ctx>(cb: EachCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): void;
+ each<Ctx>(dims: DimensionLoose, cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose], cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: EachCb2<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: ItrParamDims, cb: EachCb<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(
+ dims: ItrParamDims | EachCb<Ctx>,
+ cb: EachCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): void {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store.each(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ }
+ /**
+ * Data filter
+ */
+ filterSelf<Ctx>(cb: FilterCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): this;
+ filterSelf<Ctx>(dims: DimensionLoose, cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose], cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: FilterCb2<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: ItrParamDims, cb: FilterCb<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(
+ dims: ItrParamDims | FilterCb<Ctx>,
+ cb: FilterCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ // Clone first
+ this._store = this._store.clone();
+ this._store.filterSelf(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+
+ return this;
+ }
+
+ /**
+ * Select data in range. (For optimization of filter)
+ * (Manually inline code, support 5 million data filtering in data zoom.)
+ */
+ selectRange(range: Record<string, [number, number]>): SeriesData {
+ 'use strict';
+
+ const innerRange: Record<number, [number, number]> = {};
+ const dims = zrUtil.keys(range);
+ const dimIndices: number[] = [];
+ zrUtil.each(dims, (dim) => {
+ const dimIdx = this._getStoreDimIndex(dim);
+ innerRange[dimIdx] = range[dim];
+ dimIndices.push(dimIdx);
+ });
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store = this._store.clone();
+ this._store.selectRange(innerRange);
+ return this;
+ }
+
+ /**
+ * Data mapping to a plain array
+ */
+ mapArray<Ctx, Cb extends MapArrayCb0<Ctx>>(cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-disable */
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: DimensionLoose, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: [DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb2<Ctx>>(dims: [DimensionLoose, DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb<Ctx>>(dims: ItrParamDims, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-enable */
+ mapArray<Ctx>(
+ dims: ItrParamDims | MapArrayCb<Ctx>,
+ cb: MapArrayCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): unknown[] {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ ctx = (ctx || this) as Ctx;
+
+ const result: unknown[] = [];
+ this.each(dims, function () {
+ result.push(cb && (cb as MapArrayCb<Ctx>).apply(this, arguments));
+ }, ctx);
+ return result;
+ }
+
+ /**
+ * Data mapping to a new List with given dimensions
+ */
+ map<Ctx>(dims: DimensionLoose, cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(dims: [DimensionLoose], cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ /* eslint-disable-next-line */
+ map<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: MapCb2<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(
+ dims: ItrParamDims,
+ cb: MapCb<Ctx>,
+ ctx?: Ctx,
+ ctxCompat?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || ctxCompat || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(
+ normalizeDimensions(dims), this._getStoreDimIndex, this
+ );
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ const list = cloneListForMapAndSample(this);
+ list._store = this._store.map(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ return list;
+ }
+
+ /**
+ * Large data down sampling on given dimension
+ * @param sampleIndex Sample index for name and id
+ */
+ downSample(
Review comment:
Good point. It will be much simpler if we can reuse the same Data instance.
But for `DataStorage`. I'm thinking it should be the opposite. That all methods should be immutable. So if in one series do filter / downSample. It won't affect other Series which share the same DataStorage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675303987
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
I split the original logic in https://github.com/apache/echarts/blob/master/src/data/helper/completeDimensions.ts#L115 into two:
https://github.com/apache/echarts/blob/3de73abe262912b928e23b6eab6ac568a67bd7d7/src/data/helper/createDimensions.ts#L156
and
https://github.com/apache/echarts/blob/3de73abe262912b928e23b6eab6ac568a67bd7d7/src/data/helper/createDimensions.ts#L139
Because I need to create and set the result dimension only when it's used.
But seems the logic is not totally equal and may have issue on the duplication name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
5. If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same data. And then a data storage created by based on s
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And alway user `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like dimensionDefineList, whether dimension omitted, source for this series).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source for "omitted dimension". If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING:
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r684813263
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
The index of `SeriesData['dimensions']` and `SeriesData['getDimension']` is not the dimensionIndex that users known, but the internal dimension index if dimension count > 30.
But there are someplace output that index to users or use them directly on `SeriesData['dimensions']`:
+ in `LineView` L216 ``:
```js
const dimIndex = visualMetaList[i].dimension; // This is storage dimension index.
const dimName = data.dimensions[dimIndex]; // should not use it on `SeriesData['dimensions']`.
```
+ in `CustomView`: `wrapEncodeDef(customSeries.getData())`),
or use user input dimension index on `SeriesData['dimensions']`/`SeriesData['getDimension']`:
+ in `Customview`: `value(dim?: DimensionLoose, ...)`
+ in `component/marker/markerHelper.ts` L154 `data.getDimension(item.valueIndex)`. The valueIndex is specified by users.
+ in `component/visualMap/VisualMapModel.ts` L 390 `return data.getDimension(optDim);`.
+ ... (some other places that `SeriesData['getDimension']` is used).
Some bugs that we already have before this PR:
+ `RadarView` has bug if use `encode` before this PR:
```js
encode: {
value: [1, 2, 3, 4, 5, 6],
itemName: 7
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r684813263
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
The index of `SeriesData['dimensions']` and `SeriesData['getDimension']` is not the dimensionIndex that users known, but the internal dimension index if dimension count > 30.
But there are someplace output that index to users:
+ in `CustomView`: `wrapEncodeDef(customSeries.getData())`),
or use user input dimension index on `SeriesData['dimensions']`/`SeriesData['getDimension']`:
+ in `Customview`: `value(dim?: DimensionLoose, ...)`
+ in `component/marker/markerHelper.ts` L154 `data.getDimension(item.valueIndex)`. The valueIndex is specified by users.
+ in `component/visualMap/VisualMapModel.ts` L 390 `return data.getDimension(optDim);`.
+ ... (other place that `SeriesData['getDimension']` is used).
Some bugs that we already have before this PR:
+ `RadarView` has bug if use `encode` before this PR:
```js
encode: {
value: [1, 2, 3, 4, 5, 6],
itemName: 7
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made from 2bf188fca to 39c71111c
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And always use `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDataSchema` for each series. It contains the info generated by `createDimension` (like (1) dimensionDefineList, (2) whether dimension omitted, (3) source for this series, (4) ...).
3.1. `sourceManager` use `seriesDataSchema` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDataSchema`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDataSchema`.
3.3. `seriesData` use `seriesDataSchema` to init its dimensions, and use `seriesDataSchema` query dimName by dimIndex from source, or query dimIndex by dimName from source (only when "dimension omit" happen). If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDataSchema` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDataSchema` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
## About `SeriesDataSchema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r684813263
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
The index of `SeriesData['dimensions']` and `SeriesData['getDimension']` is not the dimensionIndex that users known, but the internal dimension index if dimension count > 30.
But there are someplace still output that index to users:
+ in `CustomView`: `wrapEncodeDef(customSeries.getData())`),
or use user input dimension index on `SeriesData['dimensions']`/`SeriesData['getDimension']`:
+ d
And `RadarView` has bug if use `encode` before this PR:
```js
encode: {
value: [1, 2, 3, 4, 5, 6],
itemName: 7
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r674490745
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
> Do we need to distinguish user defined string 'null' or 'undefined' from null / undefined here?
I don't understand what this mean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675120201
##########
File path: src/data/DataStorage.ts
##########
@@ -0,0 +1,1281 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+import { assert, clone, isFunction, keys, map, reduce } from 'zrender/src/core/util';
+import {
+ Dictionary,
+ DimensionIndex,
+ DimensionName,
+ OptionDataItem,
+ ParsedValue,
+ ParsedValueNumeric
+} from '../util/types';
+import { DataProvider } from './helper/dataProvider';
+import { parseDataValue } from './helper/dataValueHelper';
+import OrdinalMeta from './OrdinalMeta';
+import { Source } from './Source';
+
+const UNDEFINED = 'undefined';
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+// Caution: MUST not use `new CtorUint32Array(arr, 0, len)`, because the Ctor of array is
+// different from the Ctor of typed array.
+export const CtorUint32Array = typeof Uint32Array === UNDEFINED ? Array : Uint32Array;
+export const CtorUint16Array = typeof Uint16Array === UNDEFINED ? Array : Uint16Array;
+export const CtorInt32Array = typeof Int32Array === UNDEFINED ? Array : Int32Array;
+export const CtorFloat64Array = typeof Float64Array === UNDEFINED ? Array : Float64Array;
+/**
+ * Multi dimensional data storage
+ */
+const dataCtors = {
+ 'float': CtorFloat64Array,
+ 'int': CtorInt32Array,
+ // Ordinal data type can be string or int
+ 'ordinal': Array,
+ 'number': Array,
+ 'time': CtorFloat64Array
+} as const;
+
+// Dim with same category can be convert between.
+const dataCtorCategory = {
+ 'float': 0, 'time': 0,
+ 'number': 1, 'int': 2,
+ 'ordinal': 3
+};
+
+export type DataStorageDimensionType = keyof typeof dataCtors;
+
+type DataTypedArray = Uint32Array | Int32Array | Uint16Array | Float64Array;
+type DataTypedArrayConstructor = typeof Uint32Array | typeof Int32Array | typeof Uint16Array | typeof Float64Array;
+type DataArrayLikeConstructor = typeof Array | DataTypedArrayConstructor;
+
+
+type DataValueChunk = ArrayLike<ParsedValue>;
+
+// If Ctx not specified, use List as Ctx
+type EachCb0 = (idx: number) => void;
+type EachCb1 = (x: ParsedValue, idx: number) => void;
+type EachCb2 = (x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb = (...args: any) => void;
+type FilterCb0 = (idx: number) => boolean;
+type FilterCb1 = (x: ParsedValue, idx: number) => boolean;
+type FilterCb = (...args: any) => boolean;
+type MapArrayCb = (...args: any) => any;
+type MapCb = (...args: any) => ParsedValue | ParsedValue[];
+
+export type DimValueGetter = (
+ this: DataStorage,
+ dataItem: any,
+ dimName: string,
+ dataIndex: number,
+ dimIndex: DimensionIndex
+) => ParsedValue;
+
+export interface DataStorageDimensionInfo {
+ type?: DataStorageDimensionType; // Default to be float.
+ name?: string;
+ /**
+ * When using category axis.
+ * Category strings will be collected and stored in ordinalMeta.categories.
+ * And storage will store the index of categories.
+ */
+ ordinalMeta?: OrdinalMeta,
+
+ /**
+ * Offset for ordinal parsing and collect
+ */
+ ordinalOffset?: number
+}
+
+let defaultDimValueGetters: {[sourceFormat: string]: DimValueGetter};
+
+function getIndicesCtor(rawCount: number): DataArrayLikeConstructor {
+ // The possible max value in this._indicies is always this._rawCount despite of filtering.
+ return rawCount > 65535 ? CtorUint32Array : CtorUint16Array;
+};
+function getInitialExtent(): [number, number] {
+ return [Infinity, -Infinity];
+};
+function cloneChunk(originalChunk: DataValueChunk): DataValueChunk {
+ const Ctor = originalChunk.constructor;
+ // Only shallow clone is enough when Array.
+ return Ctor === Array
+ ? (originalChunk as Array<ParsedValue>).slice()
+ : new (Ctor as DataTypedArrayConstructor)(originalChunk as DataTypedArray);
+}
+
+function prepareStorage(
+ storage: DataValueChunk[],
+ dimIdx: number,
+ dimType: DataStorageDimensionType,
+ end: number,
+ append?: boolean
+): void {
+ const DataCtor = dataCtors[dimType || 'float'];
+
+ if (append) {
+ const oldStore = storage[dimIdx];
+ const oldLen = oldStore && oldStore.length;
+ if (!(oldLen === end)) {
+ const newStore = new DataCtor(end);
+ // The cost of the copy is probably inconsiderable
+ // within the initial chunkSize.
+ for (let j = 0; j < oldLen; j++) {
+ newStore[j] = oldStore[j];
+ }
+ storage[dimIdx] = newStore;
+ }
+ }
+ else {
+ storage[dimIdx] = new DataCtor(end);
+ }
+};
+class DataStorage {
+ private _chunks: DataValueChunk[] = [];
+
+ private _provider: DataProvider;
+
+ // Raw extent will not be cloned, but only transfered.
+ // It will not be calculated util needed.
+ private _rawExtent: [number, number][] = [];
+
+ private _extent: [number, number][] = [];
+
+ // Indices stores the indices of data subset after filtered.
+ // This data subset will be used in chart.
+ private _indices: ArrayLike<any>;
+
+ private _count: number = 0;
+ private _rawCount: number = 0;
+
+ private _dimensions: DataStorageDimensionInfo[];
+ private _dimValueGetter: DimValueGetter;
+
+ defaultDimValueGetter: DimValueGetter;
+ /**
+ * Initialize from data
+ * @param data source or data or data provider.
+ */
+ initData(
+ provider: DataProvider,
+ dimensions: DataStorageDimensionInfo[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ if (__DEV__) {
+ assert(
+ isFunction(provider.getItem) && isFunction(provider.count),
+ 'Inavlid data provider.'
+ );
+ }
+
+ this._provider = provider;
+
+ // Clear
+ this._chunks = [];
+ this._indices = null;
+ this.getRawIndex = this._getRawIdxIdentity;
+
+ const defaultGetter = this.defaultDimValueGetter =
+ defaultDimValueGetters[provider.getSource().sourceFormat];
+ // Default dim value getter
+ this._dimValueGetter = dimValueGetter || defaultGetter;
+
+ const emptyObj = {};
+
+ // Reset raw extent.
+ this._rawExtent = [];
+ this._dimensions = map(dimensions, dim => ({
+ // Only pick these two props. Not leak other properties like orderMeta.
+ type: dim.type,
+ name: dim.name
+ }) as DataStorageDimensionInfo);
+
+ const dimensionsIdxMap: Dictionary<number> = {};
+ let prefix = '';
+
+ // Needs to add prefix if key is used in object prototype
+ for (let i = 0; i < dimensions.length; i++) {
+ const name = dimensions[i].name;
+ if ((emptyObj as any)[name] != null) {
+ prefix = '$$';
+ }
+ }
+ for (let i = 0; i < dimensions.length; i++) {
+ const dim = dimensions[i];
+ const name = dim.name;
+ dimensionsIdxMap[prefix + name] = i;
+ }
+
+ // We use different functions because it may be a hotspot code.
+ this.getDimensionIndex = prefix ? function (dim) {
+ return dimensionsIdxMap[prefix + dim];
+ } : function (dim) {
+ return dimensionsIdxMap[dim];
+ };
+
+ this._initDataFromProvider(0, provider.count());
+ }
+
+ getProvider(): DataProvider {
+ return this._provider;
+ }
+
+ getSource(): Source {
+ return this._provider.getSource();
+ }
+
+ getDimensionIndex: (dim: DimensionName) => number;
+
+ getDimensionCount() {
+ return this._dimensions.length;
+ }
+
+ collectOrdinalMeta(
+ dimIdx: number,
+ ordinalMeta: OrdinalMeta
+ ) {
+ const chunk = this._chunks[dimIdx];
+ const dim = this._dimensions[dimIdx];
+ const rawExtents = this._rawExtent;
+
+ const offset = dim.ordinalOffset || 0;
+ const len = chunk.length;
+
+ if (offset === 0) {
+ // We need to reset the rawExtent if collect is from start.
+ // Because this dimension may be guessed as number and calcuating a wrong extent.
+ rawExtents[dimIdx] = getInitialExtent();
+ }
+
+ const dimRawExtent = rawExtents[dimIdx];
+
+ // Parse from previous data offset. len may be changed after appendData
+ for (let i = offset; i < len; i++) {
+ const val = (chunk as any)[i] = ordinalMeta.parseAndCollect(chunk[i]);
+ dimRawExtent[0] = Math.min(val, dimRawExtent[0]);
+ dimRawExtent[1] = Math.max(val, dimRawExtent[1]);
+ }
+
+ dim.ordinalMeta = ordinalMeta;
+ dim.ordinalOffset = len;
+ dim.type = 'ordinal'; // Force to be ordinal
+ }
+
+ getOrdinalMeta(dimIdx: number) {
+ const dimInfo = this._dimensions[dimIdx];
+ const ordinalMeta = dimInfo.ordinalMeta;
+ return ordinalMeta;
+ }
+
+ /**
+ * If we using dataset.
+ * Dimensions types may only know when we initializing series.
+ * So we need to sync the type back to storage when initlializing SeriesData back
+ *
+ * Will return false if dimension type has been known and different from given.
+ * We need to recreate a new data storage in this case.
+ */
+ // TODO Can't sure what's frequency will this validate fail and cause datastorage recreate.
+ syncDimensionTypes(targetDims: DataStorageDimensionInfo[]) {
+ for (let i = 0; i < targetDims.length; i++) {
+ const targetDim = targetDims[i];
+ const selfDimIdx = this.getDimensionIndex(targetDim.name);
+ const selfDim = this._dimensions[selfDimIdx];
+ if (
+ !selfDim
+ // Dim type can be convert between because ctors are compatitable.
+ || dataCtorCategory[selfDim.type || 'float'] !== dataCtorCategory[targetDim.type || 'float']
+ // ordinalMeta is different. Usually being on the different axis.
+ || (selfDim.ordinalMeta && selfDim.ordinalMeta !== targetDim.ordinalMeta)
+ ) {
+ return false;
+ }
+ selfDim.type = targetDim.type;
Review comment:
Before sync fail determined in some dimension, the previous `selfDim` has been modified, that is probably bug-prone. Any of the selfDim should better be modified after the sync success determined.
##########
File path: src/data/DataStorage.ts
##########
@@ -0,0 +1,1281 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+import { assert, clone, isFunction, keys, map, reduce } from 'zrender/src/core/util';
+import {
+ Dictionary,
+ DimensionIndex,
+ DimensionName,
+ OptionDataItem,
+ ParsedValue,
+ ParsedValueNumeric
+} from '../util/types';
+import { DataProvider } from './helper/dataProvider';
+import { parseDataValue } from './helper/dataValueHelper';
+import OrdinalMeta from './OrdinalMeta';
+import { Source } from './Source';
+
+const UNDEFINED = 'undefined';
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+// Caution: MUST not use `new CtorUint32Array(arr, 0, len)`, because the Ctor of array is
+// different from the Ctor of typed array.
+export const CtorUint32Array = typeof Uint32Array === UNDEFINED ? Array : Uint32Array;
+export const CtorUint16Array = typeof Uint16Array === UNDEFINED ? Array : Uint16Array;
+export const CtorInt32Array = typeof Int32Array === UNDEFINED ? Array : Int32Array;
+export const CtorFloat64Array = typeof Float64Array === UNDEFINED ? Array : Float64Array;
+/**
+ * Multi dimensional data storage
+ */
+const dataCtors = {
+ 'float': CtorFloat64Array,
+ 'int': CtorInt32Array,
+ // Ordinal data type can be string or int
+ 'ordinal': Array,
+ 'number': Array,
+ 'time': CtorFloat64Array
+} as const;
+
+// Dim with same category can be convert between.
+const dataCtorCategory = {
+ 'float': 0, 'time': 0,
+ 'number': 1, 'int': 2,
+ 'ordinal': 3
+};
+
+export type DataStorageDimensionType = keyof typeof dataCtors;
+
+type DataTypedArray = Uint32Array | Int32Array | Uint16Array | Float64Array;
+type DataTypedArrayConstructor = typeof Uint32Array | typeof Int32Array | typeof Uint16Array | typeof Float64Array;
+type DataArrayLikeConstructor = typeof Array | DataTypedArrayConstructor;
+
+
+type DataValueChunk = ArrayLike<ParsedValue>;
+
+// If Ctx not specified, use List as Ctx
+type EachCb0 = (idx: number) => void;
+type EachCb1 = (x: ParsedValue, idx: number) => void;
+type EachCb2 = (x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb = (...args: any) => void;
+type FilterCb0 = (idx: number) => boolean;
+type FilterCb1 = (x: ParsedValue, idx: number) => boolean;
+type FilterCb = (...args: any) => boolean;
+type MapArrayCb = (...args: any) => any;
+type MapCb = (...args: any) => ParsedValue | ParsedValue[];
+
+export type DimValueGetter = (
+ this: DataStorage,
+ dataItem: any,
+ dimName: string,
+ dataIndex: number,
+ dimIndex: DimensionIndex
+) => ParsedValue;
+
+export interface DataStorageDimensionInfo {
+ type?: DataStorageDimensionType; // Default to be float.
+ name?: string;
+ /**
+ * When using category axis.
+ * Category strings will be collected and stored in ordinalMeta.categories.
+ * And storage will store the index of categories.
+ */
+ ordinalMeta?: OrdinalMeta,
+
+ /**
+ * Offset for ordinal parsing and collect
+ */
+ ordinalOffset?: number
+}
+
+let defaultDimValueGetters: {[sourceFormat: string]: DimValueGetter};
+
+function getIndicesCtor(rawCount: number): DataArrayLikeConstructor {
+ // The possible max value in this._indicies is always this._rawCount despite of filtering.
+ return rawCount > 65535 ? CtorUint32Array : CtorUint16Array;
+};
+function getInitialExtent(): [number, number] {
+ return [Infinity, -Infinity];
+};
+function cloneChunk(originalChunk: DataValueChunk): DataValueChunk {
+ const Ctor = originalChunk.constructor;
+ // Only shallow clone is enough when Array.
+ return Ctor === Array
+ ? (originalChunk as Array<ParsedValue>).slice()
+ : new (Ctor as DataTypedArrayConstructor)(originalChunk as DataTypedArray);
+}
+
+function prepareStorage(
+ storage: DataValueChunk[],
+ dimIdx: number,
+ dimType: DataStorageDimensionType,
+ end: number,
+ append?: boolean
+): void {
+ const DataCtor = dataCtors[dimType || 'float'];
+
+ if (append) {
+ const oldStore = storage[dimIdx];
+ const oldLen = oldStore && oldStore.length;
+ if (!(oldLen === end)) {
+ const newStore = new DataCtor(end);
+ // The cost of the copy is probably inconsiderable
+ // within the initial chunkSize.
+ for (let j = 0; j < oldLen; j++) {
+ newStore[j] = oldStore[j];
+ }
+ storage[dimIdx] = newStore;
+ }
+ }
+ else {
+ storage[dimIdx] = new DataCtor(end);
+ }
+};
+class DataStorage {
+ private _chunks: DataValueChunk[] = [];
+
+ private _provider: DataProvider;
+
+ // Raw extent will not be cloned, but only transfered.
Review comment:
This comment line can be removed.
##########
File path: src/chart/helper/createSeriesDataFromArray.ts
##########
@@ -0,0 +1,208 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+import * as zrUtil from 'zrender/src/core/util';
+import SeriesData from '../../data/SeriesData';
+import createDimensions, { CreateDimensionsParams, getDimCount } from '../../data/helper/createDimensions';
+import {getDimensionTypeByAxis} from '../../data/helper/dimensionHelper';
+import {getDataItemValue} from '../../util/model';
+import CoordinateSystem from '../../core/CoordinateSystem';
+import {getCoordSysInfoBySeries} from '../../model/referHelper';
+import { createSourceFromSeriesDataOption, isSourceInstance, Source } from '../../data/Source';
+import {enableDataStack} from '../../data/helper/dataStackHelper';
+import {makeSeriesEncodeForAxisCoordSys} from '../../data/helper/sourceHelper';
+import {
+ SOURCE_FORMAT_ORIGINAL, DimensionDefinitionLoose, DimensionDefinition, OptionSourceData, EncodeDefaulter
+} from '../../util/types';
+import SeriesModel from '../../model/Series';
+import DataStorage from '../../data/DataStorage';
+import DataDimensionInfo from '../../data/DataDimensionInfo';
+
+function isDataStorage(val: unknown): val is DataStorage {
+ return val instanceof DataStorage;
+}
+
+function getCoordSysDimDefs(
+ seriesModel: SeriesModel,
+ coordSysInfo: ReturnType<typeof getCoordSysInfoBySeries>
+) {
+ const coordSysName = seriesModel.get('coordinateSystem');
+ const registeredCoordSys = CoordinateSystem.get(coordSysName);
+
+ let coordSysDimDefs: DimensionDefinitionLoose[];
+
+ if (coordSysInfo && coordSysInfo.coordSysDims) {
+ coordSysDimDefs = zrUtil.map(coordSysInfo.coordSysDims, function (dim) {
+ const dimInfo = {
+ name: dim
+ } as DimensionDefinition;
+ const axisModel = coordSysInfo.axisMap.get(dim);
+ if (axisModel) {
+ const axisType = axisModel.get('type');
+ dimInfo.type = getDimensionTypeByAxis(axisType);
+ }
+ return dimInfo;
+ });
+ }
+
+ if (!coordSysDimDefs) {
+ // Get dimensions from registered coordinate system
+ coordSysDimDefs = (registeredCoordSys && (
+ registeredCoordSys.getDimensionsInfo
+ ? registeredCoordSys.getDimensionsInfo()
+ : registeredCoordSys.dimensions.slice()
+ )) || ['x', 'y'];
+ }
+
+ return coordSysDimDefs;
+}
+
+function injectOrdinalMeta(
+ dimInfoList: DataDimensionInfo[],
+ createInvertedIndices: boolean,
+ coordSysInfo: ReturnType<typeof getCoordSysInfoBySeries>
+) {
+ let firstCategoryDimIndex: number;
+ let hasNameEncode: boolean;
+ coordSysInfo && zrUtil.each(dimInfoList, function (dimInfo, dimIndex) {
+ const coordDim = dimInfo.coordDim;
+ const categoryAxisModel = coordSysInfo.categoryAxisMap.get(coordDim);
+ if (categoryAxisModel) {
+ if (firstCategoryDimIndex == null) {
+ firstCategoryDimIndex = dimIndex;
+ }
+ dimInfo.ordinalMeta = categoryAxisModel.getOrdinalMeta();
+ if (createInvertedIndices) {
+ dimInfo.createInvertedIndices = true;
+ }
+ }
+ if (dimInfo.otherDims.itemName != null) {
+ hasNameEncode = true;
+ }
+ });
+ if (!hasNameEncode && firstCategoryDimIndex != null) {
+ dimInfoList[firstCategoryDimIndex].otherDims.itemName = 0;
+ }
+ return firstCategoryDimIndex;
+}
+
+function createListFromArray(
+ sourceOrStore: Source | OptionSourceData | DataStorage,
+ seriesModel: SeriesModel,
+ opt?: {
+ generateCoord?: string
+ useEncodeDefaulter?: boolean | EncodeDefaulter
+ // By default: auto. If `true`, create inverted indices for all ordinal dimension on coordSys.
+ createInvertedIndices?: boolean
+ }
+): SeriesData {
+ opt = opt || {};
+
+ if (!isSourceInstance(sourceOrStore) && !isDataStorage(sourceOrStore)) {
+ sourceOrStore = createSourceFromSeriesDataOption(
+ sourceOrStore as OptionSourceData
+ );
+ }
+
+ const source = isDataStorage(sourceOrStore) ? sourceOrStore.getSource() : sourceOrStore;
+ const coordSysInfo = getCoordSysInfoBySeries(seriesModel);
+ const coordSysDimDefs = getCoordSysDimDefs(seriesModel, coordSysInfo);
+ const useEncodeDefaulter = opt.useEncodeDefaulter;
+
+ // Try to ignore unsed dimensions if sharing a high dimension datastorage
+ // 30 is an experience value.
+ const omitUnusedDimensions = isDataStorage(sourceOrStore) && sourceOrStore.getDimensionCount() > 30;
+ const encodeDefaulter = zrUtil.isFunction(useEncodeDefaulter)
+ ? useEncodeDefaulter
+ : useEncodeDefaulter
+ ? zrUtil.curry(makeSeriesEncodeForAxisCoordSys, coordSysDimDefs, seriesModel)
+ : null;
+
+ const createDimensionOptions = {
+ coordDimensions: coordSysDimDefs,
+ generateCoord: opt.generateCoord,
+ encodeDefine: seriesModel.getEncode()
+ // NOTE: If we call createDimensions on same source multiple times.
+ // It will break the encodeDefaulter which has sideeffects.
+ // So we prepare the default encode here instead of passing encoderDefaulter function.
+ || (encodeDefaulter && encodeDefaulter(
+ source, getDimCount(source, coordSysDimDefs, source.dimensionsDefine || [])
+ )),
+ omitUnusedDimensions
+ };
+ let dimInfoList = createDimensions(sourceOrStore, createDimensionOptions);
+ let firstCategoryDimIndex = injectOrdinalMeta(dimInfoList, opt.createInvertedIndices, coordSysInfo);
+
+ if (omitUnusedDimensions) {
+ // sourceOrStore
+ if (!(sourceOrStore as DataStorage).syncDimensionTypes(dimInfoList)) {
+ // We need a full dimensions list if we want to recreate the storage.
+ // Or the storage will ignore the data of other dimensions.
+ dimInfoList = createDimensions(sourceOrStore, zrUtil.extend(createDimensionOptions, {
+ omitUnusedDimensions: true
Review comment:
```ts
zrUtil.extend(createDimensionOptions, {
omitUnusedDimensions: true // Should be false?
}
```
##########
File path: src/data/DataStorage.ts
##########
@@ -0,0 +1,1281 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+import { assert, clone, isFunction, keys, map, reduce } from 'zrender/src/core/util';
+import {
+ Dictionary,
+ DimensionIndex,
+ DimensionName,
+ OptionDataItem,
+ ParsedValue,
+ ParsedValueNumeric
+} from '../util/types';
+import { DataProvider } from './helper/dataProvider';
+import { parseDataValue } from './helper/dataValueHelper';
+import OrdinalMeta from './OrdinalMeta';
+import { Source } from './Source';
+
+const UNDEFINED = 'undefined';
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+// Caution: MUST not use `new CtorUint32Array(arr, 0, len)`, because the Ctor of array is
+// different from the Ctor of typed array.
+export const CtorUint32Array = typeof Uint32Array === UNDEFINED ? Array : Uint32Array;
+export const CtorUint16Array = typeof Uint16Array === UNDEFINED ? Array : Uint16Array;
+export const CtorInt32Array = typeof Int32Array === UNDEFINED ? Array : Int32Array;
+export const CtorFloat64Array = typeof Float64Array === UNDEFINED ? Array : Float64Array;
+/**
+ * Multi dimensional data storage
+ */
+const dataCtors = {
+ 'float': CtorFloat64Array,
+ 'int': CtorInt32Array,
+ // Ordinal data type can be string or int
+ 'ordinal': Array,
+ 'number': Array,
+ 'time': CtorFloat64Array
+} as const;
+
+// Dim with same category can be convert between.
+const dataCtorCategory = {
+ 'float': 0, 'time': 0,
+ 'number': 1, 'int': 2,
+ 'ordinal': 3
+};
+
+export type DataStorageDimensionType = keyof typeof dataCtors;
+
+type DataTypedArray = Uint32Array | Int32Array | Uint16Array | Float64Array;
+type DataTypedArrayConstructor = typeof Uint32Array | typeof Int32Array | typeof Uint16Array | typeof Float64Array;
+type DataArrayLikeConstructor = typeof Array | DataTypedArrayConstructor;
+
+
+type DataValueChunk = ArrayLike<ParsedValue>;
+
+// If Ctx not specified, use List as Ctx
+type EachCb0 = (idx: number) => void;
+type EachCb1 = (x: ParsedValue, idx: number) => void;
+type EachCb2 = (x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb = (...args: any) => void;
+type FilterCb0 = (idx: number) => boolean;
+type FilterCb1 = (x: ParsedValue, idx: number) => boolean;
+type FilterCb = (...args: any) => boolean;
+type MapArrayCb = (...args: any) => any;
+type MapCb = (...args: any) => ParsedValue | ParsedValue[];
+
+export type DimValueGetter = (
+ this: DataStorage,
+ dataItem: any,
+ dimName: string,
+ dataIndex: number,
+ dimIndex: DimensionIndex
+) => ParsedValue;
+
+export interface DataStorageDimensionInfo {
+ type?: DataStorageDimensionType; // Default to be float.
+ name?: string;
+ /**
+ * When using category axis.
+ * Category strings will be collected and stored in ordinalMeta.categories.
+ * And storage will store the index of categories.
+ */
+ ordinalMeta?: OrdinalMeta,
+
+ /**
+ * Offset for ordinal parsing and collect
+ */
+ ordinalOffset?: number
+}
+
+let defaultDimValueGetters: {[sourceFormat: string]: DimValueGetter};
+
+function getIndicesCtor(rawCount: number): DataArrayLikeConstructor {
+ // The possible max value in this._indicies is always this._rawCount despite of filtering.
+ return rawCount > 65535 ? CtorUint32Array : CtorUint16Array;
+};
+function getInitialExtent(): [number, number] {
+ return [Infinity, -Infinity];
+};
+function cloneChunk(originalChunk: DataValueChunk): DataValueChunk {
+ const Ctor = originalChunk.constructor;
+ // Only shallow clone is enough when Array.
+ return Ctor === Array
+ ? (originalChunk as Array<ParsedValue>).slice()
+ : new (Ctor as DataTypedArrayConstructor)(originalChunk as DataTypedArray);
+}
+
+function prepareStorage(
+ storage: DataValueChunk[],
+ dimIdx: number,
+ dimType: DataStorageDimensionType,
+ end: number,
+ append?: boolean
+): void {
+ const DataCtor = dataCtors[dimType || 'float'];
+
+ if (append) {
+ const oldStore = storage[dimIdx];
+ const oldLen = oldStore && oldStore.length;
+ if (!(oldLen === end)) {
+ const newStore = new DataCtor(end);
+ // The cost of the copy is probably inconsiderable
+ // within the initial chunkSize.
+ for (let j = 0; j < oldLen; j++) {
+ newStore[j] = oldStore[j];
+ }
+ storage[dimIdx] = newStore;
+ }
+ }
+ else {
+ storage[dimIdx] = new DataCtor(end);
+ }
+};
+class DataStorage {
+ private _chunks: DataValueChunk[] = [];
+
+ private _provider: DataProvider;
+
+ // Raw extent will not be cloned, but only transfered.
+ // It will not be calculated util needed.
+ private _rawExtent: [number, number][] = [];
+
+ private _extent: [number, number][] = [];
+
+ // Indices stores the indices of data subset after filtered.
+ // This data subset will be used in chart.
+ private _indices: ArrayLike<any>;
+
+ private _count: number = 0;
+ private _rawCount: number = 0;
+
+ private _dimensions: DataStorageDimensionInfo[];
+ private _dimValueGetter: DimValueGetter;
+
+ defaultDimValueGetter: DimValueGetter;
+ /**
+ * Initialize from data
+ * @param data source or data or data provider.
+ */
+ initData(
+ provider: DataProvider,
+ dimensions: DataStorageDimensionInfo[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ if (__DEV__) {
+ assert(
+ isFunction(provider.getItem) && isFunction(provider.count),
+ 'Inavlid data provider.'
+ );
+ }
+
+ this._provider = provider;
+
+ // Clear
+ this._chunks = [];
+ this._indices = null;
+ this.getRawIndex = this._getRawIdxIdentity;
+
+ const defaultGetter = this.defaultDimValueGetter =
+ defaultDimValueGetters[provider.getSource().sourceFormat];
+ // Default dim value getter
+ this._dimValueGetter = dimValueGetter || defaultGetter;
+
+ const emptyObj = {};
+
+ // Reset raw extent.
+ this._rawExtent = [];
+ this._dimensions = map(dimensions, dim => ({
+ // Only pick these two props. Not leak other properties like orderMeta.
+ type: dim.type,
+ name: dim.name
+ }) as DataStorageDimensionInfo);
+
+ const dimensionsIdxMap: Dictionary<number> = {};
+ let prefix = '';
+
+ // Needs to add prefix if key is used in object prototype
+ for (let i = 0; i < dimensions.length; i++) {
+ const name = dimensions[i].name;
+ if ((emptyObj as any)[name] != null) {
+ prefix = '$$';
+ }
+ }
+ for (let i = 0; i < dimensions.length; i++) {
+ const dim = dimensions[i];
+ const name = dim.name;
+ dimensionsIdxMap[prefix + name] = i;
+ }
+
+ // We use different functions because it may be a hotspot code.
+ this.getDimensionIndex = prefix ? function (dim) {
+ return dimensionsIdxMap[prefix + dim];
+ } : function (dim) {
+ return dimensionsIdxMap[dim];
+ };
+
+ this._initDataFromProvider(0, provider.count());
+ }
+
+ getProvider(): DataProvider {
+ return this._provider;
+ }
+
+ getSource(): Source {
+ return this._provider.getSource();
+ }
+
+ getDimensionIndex: (dim: DimensionName) => number;
+
+ getDimensionCount() {
+ return this._dimensions.length;
+ }
+
+ collectOrdinalMeta(
+ dimIdx: number,
+ ordinalMeta: OrdinalMeta
+ ) {
+ const chunk = this._chunks[dimIdx];
+ const dim = this._dimensions[dimIdx];
+ const rawExtents = this._rawExtent;
+
+ const offset = dim.ordinalOffset || 0;
+ const len = chunk.length;
+
+ if (offset === 0) {
+ // We need to reset the rawExtent if collect is from start.
+ // Because this dimension may be guessed as number and calcuating a wrong extent.
+ rawExtents[dimIdx] = getInitialExtent();
+ }
+
+ const dimRawExtent = rawExtents[dimIdx];
+
+ // Parse from previous data offset. len may be changed after appendData
+ for (let i = offset; i < len; i++) {
+ const val = (chunk as any)[i] = ordinalMeta.parseAndCollect(chunk[i]);
+ dimRawExtent[0] = Math.min(val, dimRawExtent[0]);
+ dimRawExtent[1] = Math.max(val, dimRawExtent[1]);
+ }
+
+ dim.ordinalMeta = ordinalMeta;
+ dim.ordinalOffset = len;
+ dim.type = 'ordinal'; // Force to be ordinal
+ }
+
+ getOrdinalMeta(dimIdx: number) {
+ const dimInfo = this._dimensions[dimIdx];
+ const ordinalMeta = dimInfo.ordinalMeta;
+ return ordinalMeta;
+ }
+
+ /**
+ * If we using dataset.
+ * Dimensions types may only know when we initializing series.
+ * So we need to sync the type back to storage when initlializing SeriesData back
+ *
+ * Will return false if dimension type has been known and different from given.
+ * We need to recreate a new data storage in this case.
+ */
+ // TODO Can't sure what's frequency will this validate fail and cause datastorage recreate.
Review comment:
### Strategy thinking
If a category dimension is used in more than one axis, for example two or more grid layout vertically where there is multiple xAxis but they are totally sync and using the same dimension, the entire dataset will not be shared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673627329
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `sysDims`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimsDef` and `opt.encodeDef`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
Review comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r684813263
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
The index of `SeriesData['dimensions']` and `SeriesData['getDimension']` is not the dimensionIndex that users known, but the internal dimension index if dimension count > 30.
But there are someplace still output that index to users:
+ in `CustomView`: `wrapEncodeDef(customSeries.getData())`),
or use user input dimension index on `SeriesData['dimensions']`/`SeriesData['getDimension']`:
+ d
And `RadarView` has bug if use `encode` before this PR.
```js
```
const userParams: CustomSeriesRenderItemParams = {
// The life cycle of context: current round of rendering.
// The global life cycle is probably not necessary, because
// user can store global status by themselves.
context: {},
seriesId: customSeries.id,
seriesName: customSeries.name,
seriesIndex: customSeries.seriesIndex,
coordSys: prepareResult.coordSys,
dataInsideLength: data.count(),
encode: wrapEncodeDef(customSeries.getData())
} as CustomSeriesRenderItemParams;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675316156
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
Review comment:
Reversed this change. `genName` should not be a performance issue if we omit the unused dimensions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
5. If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same data. And then a data storage created by based on s
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And alway user `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like dimensionDefineList, whether dimension omitted, source for this series).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source for "omitted dimension". If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675134655
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
Use this option:
```js
option = {
dataset: {
source: [
[12, 55, 33],
[11, 85, 77]
]
},
xAxis: {},
yAxis: {},
series: [{
type: 'scatter',
encode: {x: 0, y: 1},
}, {
type: 'scatter',
encode: {x: 0, y: 2},
}]
};
```
`dimensionsDefine` is `undefined` here. But I've not dig into what's wrong with it yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r674500747
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
In this case. Dataset will still generate `dimensionsDefine` for the source. `dimensionsDefine` is only undefined when the source is `series.data` instead of dataset. In which case sharing storage is meaningless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r684813263
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
The index of `SeriesData['dimensions']` and `SeriesData['getDimension']` is not the dimensionIndex that users known, but the internal dimension index if dimension count > 30.
But there are someplace output that index to users or use them directly on `SeriesData['dimensions']`:
+ in `CustomView`: `wrapEncodeDef(customSeries.getData())`),
or use user input dimension index on `SeriesData['dimensions']`/`SeriesData['getDimension']`:
+ in `Customview`: `value(dim?: DimensionLoose, ...)`
+ in `component/marker/markerHelper.ts` L154 `data.getDimension(item.valueIndex)`. The valueIndex is specified by users.
+ in `component/visualMap/VisualMapModel.ts` L 390 `return data.getDimension(optDim);`.
+ ... (other place that `SeriesData['getDimension']` is used).
Some bugs that we already have before this PR:
+ `RadarView` has bug if use `encode` before this PR:
```js
encode: {
value: [1, 2, 3, 4, 5, 6],
itemName: 7
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675303987
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
I split the original logic in https://github.com/apache/echarts/blob/master/src/data/helper/completeDimensions.ts#L115 into two:
https://github.com/apache/echarts/blob/3de73abe262912b928e23b6eab6ac568a67bd7d7/src/data/helper/createDimensions.ts#L156
and
https://github.com/apache/echarts/blob/3de73abe262912b928e23b6eab6ac568a67bd7d7/src/data/helper/createDimensions.ts#L139
Because I need to create and set the result dimension only when it's used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] echarts-bot[bot] commented on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
echarts-bot[bot] commented on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-903267022
Congratulations! Your PR has been merged. Thanks for your contribution! 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673622165
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,79 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(endSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+
+ const source = this.getSource(sourceIndex);
+ const storeList = this._storeList;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this two props as key. Another fact `dimensions` will be checked when initializing SeriesData.
+ const sourceToInit = (endSource || source);
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(endSource, sourceReadKey);
+ }
+ else {
+ const dimensionsDefine = source.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (source && dimensionsDefine) {
Review comment:
Yes, the code here may mislead the readers and it should be endSource. I've changed it a bit.
Basically, we want the storage to be always created from the source of series, which has the most correct read settings and managed/shared in the upstream source manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r674497768
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
Only in `SeriesData` will omit the unused dimensions. The `cloneAllDimensionInfo` used in data transform has no relationship with it. It uses the `dimension define from the source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675299767
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
My mistake, seems `dimensionDefine` will be undefined if `dimensions` is not given. This case seems the dimensions of these two series have the same name. Can we always generate dimensions names for dataset?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And always use `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like (1) dimensionDefineList, (2) whether dimension omitted, (3) source for this series, (4) ...).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source (only when "dimension omit" happen). If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673622165
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,79 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(endSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+
+ const source = this.getSource(sourceIndex);
+ const storeList = this._storeList;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this two props as key. Another fact `dimensions` will be checked when initializing SeriesData.
+ const sourceToInit = (endSource || source);
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(endSource, sourceReadKey);
+ }
+ else {
+ const dimensionsDefine = source.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (source && dimensionsDefine) {
Review comment:
Yes, the code here may mislead the readers and it should be endSource. I've changed it a bit.
Basically, we want the storage to be always created from the source of series, which has the most correct read settings and managed/shared in the upstream source manager so we can reduce the cost of creating data storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675393751
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
<del>I add default dimensions generation in the dataset. It seems more reasonable than picking the dimensions from the coordinate system from the dataset side. But for better display in the tooltip. it will still use `displayName` from coordinate system. Which is 'x', 'y', 'value', etc.</del>
Seems this doesn't work. It breaks cases of candlestick. I changed to another solution and adding dimension name to the key https://github.com/apache/echarts/blob/9d6c5974050e6bfb4223a19a4f4ae111c3d95b16/src/data/helper/sourceManager.ts#L531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And alway user `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like dimensionDefineList, whether dimension omitted, source for this series).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source for "omitted dimension". If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673737621
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,79 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
Review comment:
Oh, got it.
Do we need to add some brief comment or the link of this explantation in source code? Other one may not get the reason of this consideration when reading code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675324563
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
Add a general name duplication remove pass in https://github.com/apache/echarts/blob/690647c7480964e600c0fecfde52738e86f18c93/src/data/helper/createDimensions.ts#L348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r681539646
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
With the same option above,
In 5.1.2:
<img width="301" alt="Screen Shot 2021-08-03 at 4 14 20 PM" src="https://user-images.githubusercontent.com/1956569/127981913-1f307c09-a3a9-49db-820d-915625c86bb8.png">
But in the newest commit
ae9226ecc01eb60e5662f34c8bcf275dbece17d8 [ae9226ecc]
July 27, 2021 at 5:16:25 PM GMT+8
<img width="235" alt="Screen Shot 2021-08-03 at 4 16 21 PM" src="https://user-images.githubusercontent.com/1956569/127982144-e472f27a-ac12-4d35-8936-c68eacf9c280.png">
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
With the same option above,
In 5.1.2:
<img width="301" alt="Screen Shot 2021-08-03 at 4 14 20 PM" src="https://user-images.githubusercontent.com/1956569/127981913-1f307c09-a3a9-49db-820d-915625c86bb8.png">
But in the newest commit
ae9226ecc01eb60e5662f34c8bcf275dbece17d8
July 27, 2021 at 5:16:25 PM GMT+8
<img width="235" alt="Screen Shot 2021-08-03 at 4 16 21 PM" src="https://user-images.githubusercontent.com/1956569/127982144-e472f27a-ac12-4d35-8936-c68eacf9c280.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r684813263
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
The index of `SeriesData['dimensions']` and `SeriesData['getDimension']` is not the dimensionIndex that users known, but the internal dimension index if dimension count > 30.
But there are someplace output that index to users:
+ in `CustomView`: `wrapEncodeDef(customSeries.getData())`),
or use user input dimension index on `SeriesData['dimensions']`/`SeriesData['getDimension']`:
+ in `Customview`: `value(dim?: DimensionLoose, ...)`
And `RadarView` has bug if use `encode` before this PR:
```js
encode: {
value: [1, 2, 3, 4, 5, 6],
itemName: 7
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r684813263
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
The index of `SeriesData['dimensions']` and `SeriesData['getDimension']` is not the dimensionIndex that users known, but the internal dimension index if dimension count > 30.
But there are someplace output that index to users:
+ in `CustomView`: `wrapEncodeDef(customSeries.getData())`),
or use user input dimension index on `SeriesData['dimensions']`/`SeriesData['getDimension']`:
+ in
And `RadarView` has bug if use `encode` before this PR:
```js
encode: {
value: [1, 2, 3, 4, 5, 6],
itemName: 7
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675303987
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
I split the original logic in https://github.com/apache/echarts/blob/master/src/data/helper/completeDimensions.ts#L115 into two:
https://github.com/apache/echarts/blob/3de73abe262912b928e23b6eab6ac568a67bd7d7/src/data/helper/createDimensions.ts#L159
and
https://github.com/apache/echarts/blob/3de73abe262912b928e23b6eab6ac568a67bd7d7/src/data/helper/createDimensions.ts#L139
Because I need to create and set the result dimension only when it's used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And always use `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like dimensionDefineList, whether dimension omitted, source for this series).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source for "omitted dimension". If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675506703
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
Not sure if I fixed it in the latest commits. Seems this test case works as 5.1.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah merged pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah merged pull request #15355:
URL: https://github.com/apache/echarts/pull/15355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
5. If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same data. And then a data storage created by based on s
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And alway user `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like dimensionDefineList, whether dimension omitted, source for this series).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source for "omitted dimension". If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING:
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675134655
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
After 5bc0c5119 commited,
This option broken:
```js
option = {
dataset: {
source: [
[12, 55, 33],
[11, 85, 77]
]
},
xAxis: {},
yAxis: {},
series: [{
type: 'scatter',
encode: {x: 0, y: 1},
}, {
type: 'scatter',
encode: {x: 0, y: 2},
}]
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675130549
##########
File path: src/data/SeriesData.ts
##########
@@ -0,0 +1,1338 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+/**
+ * List for data storage
+ */
+
+import * as zrUtil from 'zrender/src/core/util';
+import {PathStyleProps} from 'zrender/src/graphic/Path';
+import Model from '../model/Model';
+import DataDiffer from './DataDiffer';
+import {DataProvider, DefaultDataProvider} from './helper/dataProvider';
+import {summarizeDimensions, DimensionSummary} from './helper/dimensionHelper';
+import DataDimensionInfo from './DataDimensionInfo';
+import {ArrayLike, Dictionary, FunctionPropertyNames} from 'zrender/src/core/types';
+import Element from 'zrender/src/Element';
+import {
+ DimensionIndex, DimensionName, DimensionLoose, OptionDataItem,
+ ParsedValue, ParsedValueNumeric, DimensionUserOuput,
+ ModelOption, SeriesDataType, OptionSourceData, SOURCE_FORMAT_TYPED_ARRAY, SOURCE_FORMAT_ORIGINAL,
+ DecalObject,
+ OrdinalNumber
+} from '../util/types';
+import {convertOptionIdName, isDataItemOption} from '../util/model';
+import { setCommonECData } from '../util/innerStore';
+import type Graph from './Graph';
+import type Tree from './Tree';
+import type { VisualMeta } from '../component/visualMap/VisualMapModel';
+import {isSourceInstance, Source} from './Source';
+import { LineStyleProps } from '../model/mixin/lineStyle';
+import DataStorage, { DimValueGetter } from './DataStorage';
+
+const isObject = zrUtil.isObject;
+const map = zrUtil.map;
+
+const CtorInt32Array = typeof Int32Array === 'undefined' ? Array : Int32Array;
+
+// Use prefix to avoid index to be the same as otherIdList[idx],
+// which will cause weird udpate animation.
+const ID_PREFIX = 'e\0\0';
+
+const INDEX_NOT_FOUND = -1;
+
+type NameRepeatCount = {[name: string]: number};
+type ItrParamDims = DimensionLoose | Array<DimensionLoose>;
+// If Ctx not specified, use List as Ctx
+type CtxOrList<Ctx> = unknown extends Ctx ? SeriesData : Ctx;
+type EachCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => void;
+type EachCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => void;
+type EachCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => void;
+type FilterCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => boolean;
+type FilterCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => boolean;
+type FilterCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => boolean;
+type FilterCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => boolean;
+type MapArrayCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => any;
+type MapArrayCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => any;
+type MapArrayCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => any;
+type MapArrayCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => any;
+type MapCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => ParsedValue | ParsedValue[];
+type MapCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) =>
+ ParsedValue | ParsedValue[];
+type MapCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => ParsedValue | ParsedValue[];
+
+const TRANSFERABLE_PROPERTIES = [
+ 'hasItemOption', '_nameList', '_idList', '_invertedIndicesMap',
+ '_rawData', '_dimValueGetter',
+ '_nameDimIdx', '_idDimIdx', '_nameRepeatCount'
+];
+const CLONE_PROPERTIES = [
+ '_approximateExtent'
+];
+
+export interface DefaultDataVisual {
+ style: PathStyleProps
+ // Draw type determined which prop should be set with encoded color.
+ // It's only available on the global visual. Use getVisual('drawType') to access it.
+ // It will be set in visual/style.ts module in the first priority.
+ drawType: 'fill' | 'stroke'
+
+ symbol?: string
+ symbolSize?: number | number[]
+ symbolRotate?: number
+ symbolKeepAspect?: boolean
+ symbolOffset?: string | number | (string | number)[]
+
+ liftZ?: number
+ // For legend.
+ legendIcon?: string
+ legendLineStyle?: LineStyleProps
+
+ // visualMap will inject visualMeta data
+ visualMeta?: VisualMeta[]
+
+ // If color is encoded from palette
+ colorFromPalette?: boolean
+
+ decal?: DecalObject
+}
+
+export interface DataCalculationInfo<SERIES_MODEL> {
+ stackedDimension: string;
+ stackedByDimension: string;
+ isStackedByIndex: boolean;
+ stackedOverDimension: string;
+ stackResultDimension: string;
+ stackedOnSeries?: SERIES_MODEL;
+}
+
+// -----------------------------
+// Internal method declarations:
+// -----------------------------
+let prepareInvertedIndex: (data: SeriesData) => void;
+let getId: (data: SeriesData, rawIndex: number) => string;
+let getIdNameFromStore: (data: SeriesData, dimIdx: number, dataIdx: number) => string;
+let normalizeDimensions: (dimensions: ItrParamDims) => Array<DimensionLoose>;
+let validateDimensions: (data: SeriesData, dims: DimensionIndex[]) => void;
+let transferProperties: (target: SeriesData, source: SeriesData) => void;
+let cloneListForMapAndSample: (original: SeriesData) => SeriesData;
+let makeIdFromName: (data: SeriesData, idx: number) => void;
+
+class SeriesData<
+ HostModel extends Model = Model,
+ Visual extends DefaultDataVisual = DefaultDataVisual
+> {
+
+ readonly type = 'list';
+
+ readonly dimensions: string[];
+
+ // Infomation of each data dimension, like data type.
+ private _dimensionInfos: {[dimName: string]: DataDimensionInfo};
+
+ readonly hostModel: HostModel;
+
+ /**
+ * @readonly
+ */
+ dataType: SeriesDataType;
+
+ /**
+ * @readonly
+ * Host graph if List is used to store graph nodes / edges.
+ */
+ graph?: Graph;
+
+ /**
+ * @readonly
+ * Host tree if List is used to store tree ndoes.
+ */
+ tree?: Tree;
+
+ private _store: DataStorage;
+
+ private _nameList: string[] = [];
+ private _idList: string[] = [];
+
+ // Models of data option is stored sparse for optimizing memory cost
+ // Never used yet (not used yet).
+ // private _optionModels: Model[] = [];
+
+ // Global visual properties after visual coding
+ private _visual: Dictionary<any> = {};
+
+ // Globel layout properties.
+ private _layout: Dictionary<any> = {};
+
+ // Item visual properties after visual coding
+ private _itemVisuals: Dictionary<any>[] = [];
+
+ // Item layout properties after layout
+ private _itemLayouts: any[] = [];
+
+ // Graphic elemnents
+ private _graphicEls: Element[] = [];
+
+ // key: dim, value: extent
+ private _approximateExtent: {[dimName: string]: [number, number]} = {};
+
+ private _dimensionsSummary: DimensionSummary;
+
+ private _invertedIndicesMap: {[dimName: string]: ArrayLike<number>};
+
+ private _calculationInfo: DataCalculationInfo<HostModel> = {} as DataCalculationInfo<HostModel>;
+
+ // User output info of this data.
+ // DO NOT use it in other places!
+ // When preparing user params for user callbacks, we have
+ // to clone these inner data structures to prevent users
+ // from modifying them to effect built-in logic. And for
+ // performance consideration we make this `userOutput` to
+ // avoid clone them too many times.
+ readonly userOutput: DimensionUserOuput;
+
+ // Having detected that there is data item is non primitive type
+ // (in type `OptionDataItemObject`).
+ // Like `data: [ { value: xx, itemStyle: {...} }, ...]`
+ // At present it only happen in `SOURCE_FORMAT_ORIGINAL`.
+ hasItemOption: boolean = false;
+
+ // id or name is used on dynamic data, mapping old and new items.
+ // When generating id from name, avoid repeat.
+ private _nameRepeatCount: NameRepeatCount;
+ private _nameDimIdx: number;
+ private _idDimIdx: number;
+
+ private __wrappedMethods: string[];
+
+ // Methods that create a new list based on this list should be listed here.
+ // Notice that those method should `RETURN` the new list.
+ TRANSFERABLE_METHODS = ['cloneShallow', 'downSample', 'lttbDownSample', 'map'] as const;
+ // Methods that change indices of this list should be listed here.
+ CHANGABLE_METHODS = ['filterSelf', 'selectRange'] as const;
+ DOWNSAMPLE_METHODS = ['downSample', 'lttbDownSample'] as const;
+
+ /**
+ * @param dimensions
+ * For example, ['someDimName', {name: 'someDimName', type: 'someDimType'}, ...].
+ * Dimensions should be concrete names like x, y, z, lng, lat, angle, radius
+ */
+ constructor(dimensions: Array<string | object | DataDimensionInfo>, hostModel: HostModel) {
+ dimensions = dimensions || ['x', 'y'];
+
+ const dimensionInfos: Dictionary<DataDimensionInfo> = {};
+ const dimensionNames = [];
+ const invertedIndicesMap: Dictionary<number[]> = {};
+
+ for (let i = 0; i < dimensions.length; i++) {
+ // Use the original dimensions[i], where other flag props may exists.
+ const dimInfoInput = dimensions[i];
+
+ const dimensionInfo: DataDimensionInfo =
+ zrUtil.isString(dimInfoInput)
+ ? new DataDimensionInfo({name: dimInfoInput})
+ : !(dimInfoInput instanceof DataDimensionInfo)
+ ? new DataDimensionInfo(dimInfoInput)
+ : dimInfoInput;
+
+ const dimensionName = dimensionInfo.name;
+ dimensionInfo.type = dimensionInfo.type || 'float';
+ if (!dimensionInfo.coordDim) {
+ dimensionInfo.coordDim = dimensionName;
+ dimensionInfo.coordDimIndex = 0;
+ }
+
+ const otherDims = dimensionInfo.otherDims = dimensionInfo.otherDims || {};
+ dimensionNames.push(dimensionName);
+ dimensionInfos[dimensionName] = dimensionInfo;
+
+ dimensionInfo.index = i;
+
+ if (dimensionInfo.createInvertedIndices) {
+ invertedIndicesMap[dimensionName] = [];
+ }
+ if (otherDims.itemName === 0) {
+ this._nameDimIdx = i;
+ }
+ if (otherDims.itemId === 0) {
+ this._idDimIdx = i;
+ }
+ }
+
+ this.dimensions = dimensionNames;
+ this._dimensionInfos = dimensionInfos;
+ this.hostModel = hostModel;
+
+ // Cache summary info for fast visit. See "dimensionHelper".
+ this._dimensionsSummary = summarizeDimensions(this);
+
+ this._invertedIndicesMap = invertedIndicesMap;
+
+ this.userOutput = this._dimensionsSummary.userOutput;
+ }
+
+ /**
+ * The meanings of the input parameter `dim`:
+ *
+ * + If dim is a number (e.g., `1`), it means the index of the dimension.
+ * For example, `getDimension(0)` will return 'x' or 'lng' or 'radius'.
+ * + If dim is a number-like string (e.g., `"1"`):
+ * + If there is the same concrete dim name defined in `this.dimensions`, it means that concrete name.
+ * + If not, it will be converted to a number, which means the index of the dimension.
+ * (why? because of the backward compatbility. We have been tolerating number-like string in
+ * dimension setting, although now it seems that it is not a good idea.)
+ * For example, `visualMap[i].dimension: "1"` is the same meaning as `visualMap[i].dimension: 1`,
+ * if no dimension name is defined as `"1"`.
+ * + If dim is a not-number-like string, it means the concrete dim name.
+ * For example, it can be be default name `"x"`, `"y"`, `"z"`, `"lng"`, `"lat"`, `"angle"`, `"radius"`,
+ * or customized in `dimensions` property of option like `"age"`.
+ *
+ * Get dimension name
+ * @param dim See above.
+ * @return Concrete dim name.
+ */
+ getDimension(dim: DimensionLoose): DimensionName {
+ if (typeof dim === 'number'
+ // If being a number-like string but not being defined a dimension name.
+ || (!isNaN(dim as any) && !this._dimensionInfos.hasOwnProperty(dim))
+ ) {
+ dim = this.dimensions[dim as DimensionIndex];
+ }
+ return dim as DimensionName;
+ }
+
+ private _getStoreDimIndex(dim: DimensionLoose): DimensionIndex {
+ return this._store.getDimensionIndex(this.getDimension(dim));
+ }
+
+ /**
+ * Get type and calculation info of particular dimension
+ * @param dim
+ * Dimension can be concrete names like x, y, z, lng, lat, angle, radius
+ * Or a ordinal number. For example getDimensionInfo(0) will return 'x' or 'lng' or 'radius'
+ */
+ getDimensionInfo(dim: DimensionLoose): DataDimensionInfo {
+ // Do not clone, because there may be categories in dimInfo.
+ return this._dimensionInfos[this.getDimension(dim)];
+ }
+
+ /**
+ * concrete dimension name list on coord.
+ */
+ getDimensionsOnCoord(): DimensionName[] {
+ return this._dimensionsSummary.dataDimsOnCoord.slice();
+ }
+
+ /**
+ * @param coordDim
+ * @param idx A coordDim may map to more than one data dim.
+ * If not specified, return the first dim not extra.
+ * @return concrete data dim. If not found, return null/undefined
+ */
+ mapDimension(coordDim: DimensionName): DimensionName;
+ mapDimension(coordDim: DimensionName, idx: number): DimensionName;
+ mapDimension(coordDim: DimensionName, idx?: number): DimensionName {
+ const dimensionsSummary = this._dimensionsSummary;
+
+ if (idx == null) {
+ return dimensionsSummary.encodeFirstDimNotExtra[coordDim] as any;
+ }
+
+ const dims = dimensionsSummary.encode[coordDim];
+ return dims ? dims[idx as number] as any : null;
+ }
+
+ mapDimensionsAll(coordDim: DimensionName): DimensionName[] {
+ const dimensionsSummary = this._dimensionsSummary;
+ const dims = dimensionsSummary.encode[coordDim];
+ return (dims || []).slice();
+ }
+
+ getStorage() {
+ return this._store;
+ }
+
+ /**
+ * Initialize from data
+ * @param data source or data or data storage.
+ * @param nameList The name of a datum is used on data diff and
+ * default label/tooltip.
+ * A name can be specified in encode.itemName,
+ * or dataItem.name (only for series option data),
+ * or provided in nameList from outside.
+ */
+ initData(
+ data: Source | OptionSourceData | DataStorage | DataProvider,
+ nameList?: string[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ let store: DataStorage;
+ const dimensions = this.dimensions;
+ const dimensionInfos = map(dimensions, dimName => this._dimensionInfos[dimName]);
+ if (data instanceof DataStorage) {
+ if (data.syncDimensionTypes(dimensionInfos)) {
+ store = data;
+ }
+ // Sync failed
+ else {
+ // Needs to recreate data storage if it's not match the given dimension.
+ data = data.getSource();
+ }
+ }
+
+ if (!store) {
+ const provider = (isSourceInstance(data) || zrUtil.isArrayLike(data))
+ ? new DefaultDataProvider(data as Source | OptionSourceData, dimensions.length)
+ : data as DataProvider;
+ store = new DataStorage();
+ store.initData(provider, dimensionInfos, dimValueGetter);
+ }
+
+ this._store = store;
+
+ // Reset
+ this._nameList = (nameList || []).slice();
+ this._idList = [];
+ this._nameRepeatCount = {};
+
+ this._doInit(0, store.count());
+ }
+
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ */
+ appendData(data: ArrayLike<any>): void {
+ const range = this._store.appendData(data);
+ this._doInit(range[0], range[1]);
+ }
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ * This method does not modify `rawData` (`dataProvider`), but only
+ * add values to storage.
+ *
+ * The final count will be increased by `Math.max(values.length, names.length)`.
+ *
+ * @param values That is the SourceType: 'arrayRows', like
+ * [
+ * [12, 33, 44],
+ * [NaN, 43, 1],
+ * ['-', 'asdf', 0]
+ * ]
+ * Each item is exaclty cooresponding to a dimension.
+ */
+ appendValues(values: any[][], names?: string[]): void {
+ const {start, end} = this._store.appendValues(values, names.length);
+ const shouldMakeIdFromName = this._shouldMakeIdFromName();
+
+ this._updateOrdinalMeta();
+
+ if (names) {
+ for (let idx = start; idx < end; idx++) {
+ const sourceIdx = idx - start;
+ this._nameList[idx] = names[sourceIdx];
+ if (shouldMakeIdFromName) {
+ makeIdFromName(this, idx);
+ }
+ }
+ }
+ }
+
+ private _updateOrdinalMeta() {
+ const store = this._store;
+ const dimensions = this.dimensions;
+ for (let i = 0; i < dimensions.length; i++) {
+ const dimInfo = this._dimensionInfos[dimensions[i]];
+ if (dimInfo.ordinalMeta) {
+ const dimIdx = store.getDimensionIndex(dimensions[i]);
+ store.collectOrdinalMeta(dimIdx, dimInfo.ordinalMeta);
+ }
+ }
+ }
+
+ private _shouldMakeIdFromName() {
+ const provider = this._store.getProvider();
+ return this._idDimIdx == null
+ && provider.getSource().sourceFormat !== SOURCE_FORMAT_TYPED_ARRAY
+ && !provider.fillStorage;
+ }
+
+ private _doInit(start: number, end: number): void {
+ if (start >= end) {
+ return;
+ }
+
+ const store = this._store;
+ const provider = store.getProvider();
+
+ this._updateOrdinalMeta();
+
+ const nameList = this._nameList;
+ const idList = this._idList;
+ const sourceFormat = provider.getSource().sourceFormat;
+ const isFormatOriginal = sourceFormat === SOURCE_FORMAT_ORIGINAL;
+
+ // Each data item is value
+ // [1, 2]
+ // 2
+ // Bar chart, line chart which uses category axis
+ // only gives the 'y' value. 'x' value is the indices of category
+ // Use a tempValue to normalize the value to be a (x, y) value
+ // If dataItem is {name: ...} or {id: ...}, it has highest priority.
+ // This kind of ids and names are always stored `_nameList` and `_idList`.
+ if (isFormatOriginal && !provider.pure) {
+ const sharedDataItem = [] as OptionDataItem;
+ for (let idx = start; idx < end; idx++) {
+ // NOTICE: Try not to write things into dataItem
+ const dataItem = provider.getItem(idx, sharedDataItem);
+ if (!this.hasItemOption && isDataItemOption(dataItem)) {
+ this.hasItemOption = true;
+ }
+ if (dataItem) {
+ const itemName = (dataItem as any).name;
+ if (nameList[idx] == null && itemName != null) {
+ nameList[idx] = convertOptionIdName(itemName, null);
+ }
+ const itemId = (dataItem as any).id;
+ if (idList[idx] == null && itemId != null) {
+ idList[idx] = convertOptionIdName(itemId, null);
+ }
+ }
+ }
+ }
+
+ if (this._shouldMakeIdFromName()) {
+ for (let idx = start; idx < end; idx++) {
+ makeIdFromName(this, idx);
+ }
+ }
+
+ prepareInvertedIndex(this);
+ }
+ /**
+ * PENDING: In fact currently this function is only used to short-circuit
+ * the calling of `scale.unionExtentFromData` when data have been filtered by modules
+ * like "dataZoom". `scale.unionExtentFromData` is used to calculate data extent for series on
+ * an axis, but if a "axis related data filter module" is used, the extent of the axis have
+ * been fixed and no need to calling `scale.unionExtentFromData` actually.
+ * But if we add "custom data filter" in future, which is not "axis related", this method may
+ * be still needed.
+ *
+ * Optimize for the scenario that data is filtered by a given extent.
+ * Consider that if data amount is more than hundreds of thousand,
+ * extent calculation will cost more than 10ms and the cache will
+ * be erased because of the filtering.
+ */
+ getApproximateExtent(dim: DimensionLoose): [number, number] {
+ return this._approximateExtent[dim] || this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ /**
+ * Calculate extent on a filtered data might be time consuming.
+ * Approximate extent is only used for: calculte extent of filtered data outside.
+ */
+ setApproximateExtent(extent: [number, number], dim: DimensionLoose): void {
+ dim = this.getDimension(dim);
+ this._approximateExtent[dim] = extent.slice() as [number, number];
+ }
+
+ getCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY
+ ): DataCalculationInfo<HostModel>[CALC_INFO_KEY] {
+ return this._calculationInfo[key];
+ }
+
+ /**
+ * @param key or k-v object
+ */
+ setCalculationInfo(
+ key: DataCalculationInfo<HostModel>
+ ): void;
+ setCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY,
+ value: DataCalculationInfo<HostModel>[CALC_INFO_KEY]
+ ): void;
+ setCalculationInfo(
+ key: (keyof DataCalculationInfo<HostModel>) | DataCalculationInfo<HostModel>,
+ value?: DataCalculationInfo<HostModel>[keyof DataCalculationInfo<HostModel>]
+ ): void {
+ isObject(key)
+ ? zrUtil.extend(this._calculationInfo, key as object)
+ : ((this._calculationInfo as any)[key] = value);
+ }
+
+ /**
+ * @return Never be null/undefined. `number` will be converted to string. Becuase:
+ * In most cases, name is used in display, where returning a string is more convenient.
+ * In other cases, name is used in query (see `indexOfName`), where we can keep the
+ * rule that name `2` equals to name `'2'`.
+ */
+ getName(idx: number): string {
+ const rawIndex = this.getRawIndex(idx);
+ let name = this._nameList[rawIndex];
+ if (name == null && this._nameDimIdx != null) {
+ name = getIdNameFromStore(this, this._nameDimIdx, rawIndex);
+ }
+ if (name == null) {
+ name = '';
+ }
+ return name;
+ }
+
+ private _getCategory(dimIdx: number, idx: number) {
+ const ordinal = this._store.get(dimIdx, idx);
+ const ordinalMeta = this._store.getOrdinalMeta(dimIdx);
+ if (ordinalMeta) {
+ return ordinalMeta.categories[ordinal as OrdinalNumber];
+ }
+ return ordinal;
+ }
+
+ /**
+ * @return Never null/undefined. `number` will be converted to string. Becuase:
+ * In all cases having encountered at present, id is used in making diff comparison, which
+ * are usually based on hash map. We can keep the rule that the internal id are always string
+ * (treat `2` is the same as `'2'`) to make the related logic simple.
+ */
+ getId(idx: number): string {
+ return getId(this, this.getRawIndex(idx));
+ }
+
+ count() {
+ return this._store.count();
+ }
+ /**
+ * Get value. Return NaN if idx is out of range.
+ * @param dim Dim must be concrete name.
+ */
+ get(dim: DimensionName, idx: number): ParsedValue {
+ const store = this._store;
+ return store.get(store.getDimensionIndex(dim), idx);
+ }
+
+ getByRawIndex(dim: DimensionName, rawIdx: number): ParsedValue {
+ const store = this._store;
+ return store.getByRawIndex(store.getDimensionIndex(dim), rawIdx);
+ }
+
+ getIndices() {
+ return this._store.getIndices();
+ }
+
+ getDataExtent(dim: DimensionLoose): [number, number] {
+ return this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ getSum(dim: DimensionLoose): number {
+ return this._store.getSum(this._getStoreDimIndex(dim));
+ }
+
+ getMedian(dim: DimensionLoose): number {
+ return this._store.getMedian(this._getStoreDimIndex(dim));
+ }
+ /**
+ * Get value for multi dimensions.
+ * @param dimensions If ignored, using all dimensions.
+ */
+ getValues(idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[], idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[] | number, idx?: number): ParsedValue[] {
+ const store = this._store;
+ return zrUtil.isArray(dimensions)
+ ? store.getValues(map(dimensions, dim => this._getStoreDimIndex(dim)), idx)
+ : store.getValues(dimensions as number);
+ }
+
+ /**
+ * If value is NaN. Inlcuding '-'
+ * Only check the coord dimensions.
+ */
+ hasValue(idx: number): boolean {
+ const dataDimsOnCoord = this._dimensionsSummary.dataDimsOnCoord;
+ for (let i = 0, len = dataDimsOnCoord.length; i < len; i++) {
+ // Ordinal type originally can be string or number.
+ // But when an ordinal type is used on coord, it can
+ // not be string but only number. So we can also use isNaN.
+ if (isNaN(this.get(dataDimsOnCoord[i], idx) as any)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Retreive the index with given name
+ */
+ indexOfName(name: string): number {
+ for (let i = 0, len = this._store.count(); i < len; i++) {
+ if (this.getName(i) === name) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ getRawIndex(idx: number) {
+ return this._store.getRawIndex(idx);
+ }
+
+ indexOfRawIndex(rawIndex: number) {
+ return this._store.indexOfRawIndex(rawIndex);
+ }
+
+ /**
+ * Only support the dimension which inverted index created.
+ * Do not support other cases until required.
+ * @param dim concrete dim
+ * @param value ordinal index
+ * @return rawIndex
+ */
+ rawIndexOf(dim: DimensionName, value: OrdinalNumber): number {
+ const invertedIndices = dim && this._invertedIndicesMap[dim];
+ if (__DEV__) {
+ if (!invertedIndices) {
+ throw new Error('Do not supported yet');
+ }
+ }
+ const rawIndex = invertedIndices[value];
+ if (rawIndex == null || isNaN(rawIndex)) {
+ return INDEX_NOT_FOUND;
+ }
+ return rawIndex;
+ }
+
+ /**
+ * Retreive the index of nearest value
+ * @param dim
+ * @param value
+ * @param [maxDistance=Infinity]
+ * @return If and only if multiple indices has
+ * the same value, they are put to the result.
+ */
+ indicesOfNearest(dim: DimensionLoose, value: number, maxDistance?: number): number[] {
+ return this._store.indicesOfNearest(
+ this._getStoreDimIndex(dim),
+ value, maxDistance
+ );
+ }
+ /**
+ * Data iteration
+ * @param ctx default this
+ * @example
+ * list.each('x', function (x, idx) {});
+ * list.each(['x', 'y'], function (x, y, idx) {});
+ * list.each(function (idx) {})
+ */
+ each<Ctx>(cb: EachCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): void;
+ each<Ctx>(dims: DimensionLoose, cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose], cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: EachCb2<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: ItrParamDims, cb: EachCb<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(
+ dims: ItrParamDims | EachCb<Ctx>,
+ cb: EachCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): void {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store.each(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ }
+ /**
+ * Data filter
+ */
+ filterSelf<Ctx>(cb: FilterCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): this;
+ filterSelf<Ctx>(dims: DimensionLoose, cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose], cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: FilterCb2<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: ItrParamDims, cb: FilterCb<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(
+ dims: ItrParamDims | FilterCb<Ctx>,
+ cb: FilterCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ // Clone first
+ this._store = this._store.clone();
+ this._store.filterSelf(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+
+ return this;
+ }
+
+ /**
+ * Select data in range. (For optimization of filter)
+ * (Manually inline code, support 5 million data filtering in data zoom.)
+ */
+ selectRange(range: Record<string, [number, number]>): SeriesData {
+ 'use strict';
+
+ const innerRange: Record<number, [number, number]> = {};
+ const dims = zrUtil.keys(range);
+ const dimIndices: number[] = [];
+ zrUtil.each(dims, (dim) => {
+ const dimIdx = this._getStoreDimIndex(dim);
+ innerRange[dimIdx] = range[dim];
+ dimIndices.push(dimIdx);
+ });
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store = this._store.clone();
+ this._store.selectRange(innerRange);
+ return this;
+ }
+
+ /**
+ * Data mapping to a plain array
+ */
+ mapArray<Ctx, Cb extends MapArrayCb0<Ctx>>(cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-disable */
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: DimensionLoose, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: [DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb2<Ctx>>(dims: [DimensionLoose, DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb<Ctx>>(dims: ItrParamDims, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-enable */
+ mapArray<Ctx>(
+ dims: ItrParamDims | MapArrayCb<Ctx>,
+ cb: MapArrayCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): unknown[] {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ ctx = (ctx || this) as Ctx;
+
+ const result: unknown[] = [];
+ this.each(dims, function () {
+ result.push(cb && (cb as MapArrayCb<Ctx>).apply(this, arguments));
+ }, ctx);
+ return result;
+ }
+
+ /**
+ * Data mapping to a new List with given dimensions
+ */
+ map<Ctx>(dims: DimensionLoose, cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(dims: [DimensionLoose], cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ /* eslint-disable-next-line */
+ map<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: MapCb2<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(
+ dims: ItrParamDims,
+ cb: MapCb<Ctx>,
+ ctx?: Ctx,
+ ctxCompat?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || ctxCompat || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(
+ normalizeDimensions(dims), this._getStoreDimIndex, this
+ );
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ const list = cloneListForMapAndSample(this);
+ list._store = this._store.map(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ return list;
+ }
+
+ /**
+ * Large data down sampling on given dimension
+ * @param sampleIndex Sample index for name and id
+ */
+ downSample(
Review comment:
Yes I agree, `DataStorage` will be totally immutable. (The only exception is `appendData` and `appendValue`, which are restricted to only be called in `Series#getRawData`).
##########
File path: src/data/SeriesData.ts
##########
@@ -0,0 +1,1338 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+/**
+ * List for data storage
+ */
+
+import * as zrUtil from 'zrender/src/core/util';
+import {PathStyleProps} from 'zrender/src/graphic/Path';
+import Model from '../model/Model';
+import DataDiffer from './DataDiffer';
+import {DataProvider, DefaultDataProvider} from './helper/dataProvider';
+import {summarizeDimensions, DimensionSummary} from './helper/dimensionHelper';
+import DataDimensionInfo from './DataDimensionInfo';
+import {ArrayLike, Dictionary, FunctionPropertyNames} from 'zrender/src/core/types';
+import Element from 'zrender/src/Element';
+import {
+ DimensionIndex, DimensionName, DimensionLoose, OptionDataItem,
+ ParsedValue, ParsedValueNumeric, DimensionUserOuput,
+ ModelOption, SeriesDataType, OptionSourceData, SOURCE_FORMAT_TYPED_ARRAY, SOURCE_FORMAT_ORIGINAL,
+ DecalObject,
+ OrdinalNumber
+} from '../util/types';
+import {convertOptionIdName, isDataItemOption} from '../util/model';
+import { setCommonECData } from '../util/innerStore';
+import type Graph from './Graph';
+import type Tree from './Tree';
+import type { VisualMeta } from '../component/visualMap/VisualMapModel';
+import {isSourceInstance, Source} from './Source';
+import { LineStyleProps } from '../model/mixin/lineStyle';
+import DataStorage, { DimValueGetter } from './DataStorage';
+
+const isObject = zrUtil.isObject;
+const map = zrUtil.map;
+
+const CtorInt32Array = typeof Int32Array === 'undefined' ? Array : Int32Array;
+
+// Use prefix to avoid index to be the same as otherIdList[idx],
+// which will cause weird udpate animation.
+const ID_PREFIX = 'e\0\0';
+
+const INDEX_NOT_FOUND = -1;
+
+type NameRepeatCount = {[name: string]: number};
+type ItrParamDims = DimensionLoose | Array<DimensionLoose>;
+// If Ctx not specified, use List as Ctx
+type CtxOrList<Ctx> = unknown extends Ctx ? SeriesData : Ctx;
+type EachCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => void;
+type EachCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => void;
+type EachCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => void;
+type FilterCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => boolean;
+type FilterCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => boolean;
+type FilterCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => boolean;
+type FilterCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => boolean;
+type MapArrayCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => any;
+type MapArrayCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => any;
+type MapArrayCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => any;
+type MapArrayCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => any;
+type MapCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => ParsedValue | ParsedValue[];
+type MapCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) =>
+ ParsedValue | ParsedValue[];
+type MapCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => ParsedValue | ParsedValue[];
+
+const TRANSFERABLE_PROPERTIES = [
+ 'hasItemOption', '_nameList', '_idList', '_invertedIndicesMap',
+ '_rawData', '_dimValueGetter',
+ '_nameDimIdx', '_idDimIdx', '_nameRepeatCount'
+];
+const CLONE_PROPERTIES = [
+ '_approximateExtent'
+];
+
+export interface DefaultDataVisual {
+ style: PathStyleProps
+ // Draw type determined which prop should be set with encoded color.
+ // It's only available on the global visual. Use getVisual('drawType') to access it.
+ // It will be set in visual/style.ts module in the first priority.
+ drawType: 'fill' | 'stroke'
+
+ symbol?: string
+ symbolSize?: number | number[]
+ symbolRotate?: number
+ symbolKeepAspect?: boolean
+ symbolOffset?: string | number | (string | number)[]
+
+ liftZ?: number
+ // For legend.
+ legendIcon?: string
+ legendLineStyle?: LineStyleProps
+
+ // visualMap will inject visualMeta data
+ visualMeta?: VisualMeta[]
+
+ // If color is encoded from palette
+ colorFromPalette?: boolean
+
+ decal?: DecalObject
+}
+
+export interface DataCalculationInfo<SERIES_MODEL> {
+ stackedDimension: string;
+ stackedByDimension: string;
+ isStackedByIndex: boolean;
+ stackedOverDimension: string;
+ stackResultDimension: string;
+ stackedOnSeries?: SERIES_MODEL;
+}
+
+// -----------------------------
+// Internal method declarations:
+// -----------------------------
+let prepareInvertedIndex: (data: SeriesData) => void;
+let getId: (data: SeriesData, rawIndex: number) => string;
+let getIdNameFromStore: (data: SeriesData, dimIdx: number, dataIdx: number) => string;
+let normalizeDimensions: (dimensions: ItrParamDims) => Array<DimensionLoose>;
+let validateDimensions: (data: SeriesData, dims: DimensionIndex[]) => void;
+let transferProperties: (target: SeriesData, source: SeriesData) => void;
+let cloneListForMapAndSample: (original: SeriesData) => SeriesData;
+let makeIdFromName: (data: SeriesData, idx: number) => void;
+
+class SeriesData<
+ HostModel extends Model = Model,
+ Visual extends DefaultDataVisual = DefaultDataVisual
+> {
+
+ readonly type = 'list';
+
+ readonly dimensions: string[];
+
+ // Infomation of each data dimension, like data type.
+ private _dimensionInfos: {[dimName: string]: DataDimensionInfo};
+
+ readonly hostModel: HostModel;
+
+ /**
+ * @readonly
+ */
+ dataType: SeriesDataType;
+
+ /**
+ * @readonly
+ * Host graph if List is used to store graph nodes / edges.
+ */
+ graph?: Graph;
+
+ /**
+ * @readonly
+ * Host tree if List is used to store tree ndoes.
+ */
+ tree?: Tree;
+
+ private _store: DataStorage;
+
+ private _nameList: string[] = [];
+ private _idList: string[] = [];
+
+ // Models of data option is stored sparse for optimizing memory cost
+ // Never used yet (not used yet).
+ // private _optionModels: Model[] = [];
+
+ // Global visual properties after visual coding
+ private _visual: Dictionary<any> = {};
+
+ // Globel layout properties.
+ private _layout: Dictionary<any> = {};
+
+ // Item visual properties after visual coding
+ private _itemVisuals: Dictionary<any>[] = [];
+
+ // Item layout properties after layout
+ private _itemLayouts: any[] = [];
+
+ // Graphic elemnents
+ private _graphicEls: Element[] = [];
+
+ // key: dim, value: extent
+ private _approximateExtent: {[dimName: string]: [number, number]} = {};
+
+ private _dimensionsSummary: DimensionSummary;
+
+ private _invertedIndicesMap: {[dimName: string]: ArrayLike<number>};
+
+ private _calculationInfo: DataCalculationInfo<HostModel> = {} as DataCalculationInfo<HostModel>;
+
+ // User output info of this data.
+ // DO NOT use it in other places!
+ // When preparing user params for user callbacks, we have
+ // to clone these inner data structures to prevent users
+ // from modifying them to effect built-in logic. And for
+ // performance consideration we make this `userOutput` to
+ // avoid clone them too many times.
+ readonly userOutput: DimensionUserOuput;
+
+ // Having detected that there is data item is non primitive type
+ // (in type `OptionDataItemObject`).
+ // Like `data: [ { value: xx, itemStyle: {...} }, ...]`
+ // At present it only happen in `SOURCE_FORMAT_ORIGINAL`.
+ hasItemOption: boolean = false;
+
+ // id or name is used on dynamic data, mapping old and new items.
+ // When generating id from name, avoid repeat.
+ private _nameRepeatCount: NameRepeatCount;
+ private _nameDimIdx: number;
+ private _idDimIdx: number;
+
+ private __wrappedMethods: string[];
+
+ // Methods that create a new list based on this list should be listed here.
+ // Notice that those method should `RETURN` the new list.
+ TRANSFERABLE_METHODS = ['cloneShallow', 'downSample', 'lttbDownSample', 'map'] as const;
+ // Methods that change indices of this list should be listed here.
+ CHANGABLE_METHODS = ['filterSelf', 'selectRange'] as const;
+ DOWNSAMPLE_METHODS = ['downSample', 'lttbDownSample'] as const;
+
+ /**
+ * @param dimensions
+ * For example, ['someDimName', {name: 'someDimName', type: 'someDimType'}, ...].
+ * Dimensions should be concrete names like x, y, z, lng, lat, angle, radius
+ */
+ constructor(dimensions: Array<string | object | DataDimensionInfo>, hostModel: HostModel) {
+ dimensions = dimensions || ['x', 'y'];
+
+ const dimensionInfos: Dictionary<DataDimensionInfo> = {};
+ const dimensionNames = [];
+ const invertedIndicesMap: Dictionary<number[]> = {};
+
+ for (let i = 0; i < dimensions.length; i++) {
+ // Use the original dimensions[i], where other flag props may exists.
+ const dimInfoInput = dimensions[i];
+
+ const dimensionInfo: DataDimensionInfo =
+ zrUtil.isString(dimInfoInput)
+ ? new DataDimensionInfo({name: dimInfoInput})
+ : !(dimInfoInput instanceof DataDimensionInfo)
+ ? new DataDimensionInfo(dimInfoInput)
+ : dimInfoInput;
+
+ const dimensionName = dimensionInfo.name;
+ dimensionInfo.type = dimensionInfo.type || 'float';
+ if (!dimensionInfo.coordDim) {
+ dimensionInfo.coordDim = dimensionName;
+ dimensionInfo.coordDimIndex = 0;
+ }
+
+ const otherDims = dimensionInfo.otherDims = dimensionInfo.otherDims || {};
+ dimensionNames.push(dimensionName);
+ dimensionInfos[dimensionName] = dimensionInfo;
+
+ dimensionInfo.index = i;
+
+ if (dimensionInfo.createInvertedIndices) {
+ invertedIndicesMap[dimensionName] = [];
+ }
+ if (otherDims.itemName === 0) {
+ this._nameDimIdx = i;
+ }
+ if (otherDims.itemId === 0) {
+ this._idDimIdx = i;
+ }
+ }
+
+ this.dimensions = dimensionNames;
+ this._dimensionInfos = dimensionInfos;
+ this.hostModel = hostModel;
+
+ // Cache summary info for fast visit. See "dimensionHelper".
+ this._dimensionsSummary = summarizeDimensions(this);
+
+ this._invertedIndicesMap = invertedIndicesMap;
+
+ this.userOutput = this._dimensionsSummary.userOutput;
+ }
+
+ /**
+ * The meanings of the input parameter `dim`:
+ *
+ * + If dim is a number (e.g., `1`), it means the index of the dimension.
+ * For example, `getDimension(0)` will return 'x' or 'lng' or 'radius'.
+ * + If dim is a number-like string (e.g., `"1"`):
+ * + If there is the same concrete dim name defined in `this.dimensions`, it means that concrete name.
+ * + If not, it will be converted to a number, which means the index of the dimension.
+ * (why? because of the backward compatbility. We have been tolerating number-like string in
+ * dimension setting, although now it seems that it is not a good idea.)
+ * For example, `visualMap[i].dimension: "1"` is the same meaning as `visualMap[i].dimension: 1`,
+ * if no dimension name is defined as `"1"`.
+ * + If dim is a not-number-like string, it means the concrete dim name.
+ * For example, it can be be default name `"x"`, `"y"`, `"z"`, `"lng"`, `"lat"`, `"angle"`, `"radius"`,
+ * or customized in `dimensions` property of option like `"age"`.
+ *
+ * Get dimension name
+ * @param dim See above.
+ * @return Concrete dim name.
+ */
+ getDimension(dim: DimensionLoose): DimensionName {
+ if (typeof dim === 'number'
+ // If being a number-like string but not being defined a dimension name.
+ || (!isNaN(dim as any) && !this._dimensionInfos.hasOwnProperty(dim))
+ ) {
+ dim = this.dimensions[dim as DimensionIndex];
+ }
+ return dim as DimensionName;
+ }
+
+ private _getStoreDimIndex(dim: DimensionLoose): DimensionIndex {
+ return this._store.getDimensionIndex(this.getDimension(dim));
+ }
+
+ /**
+ * Get type and calculation info of particular dimension
+ * @param dim
+ * Dimension can be concrete names like x, y, z, lng, lat, angle, radius
+ * Or a ordinal number. For example getDimensionInfo(0) will return 'x' or 'lng' or 'radius'
+ */
+ getDimensionInfo(dim: DimensionLoose): DataDimensionInfo {
+ // Do not clone, because there may be categories in dimInfo.
+ return this._dimensionInfos[this.getDimension(dim)];
+ }
+
+ /**
+ * concrete dimension name list on coord.
+ */
+ getDimensionsOnCoord(): DimensionName[] {
+ return this._dimensionsSummary.dataDimsOnCoord.slice();
+ }
+
+ /**
+ * @param coordDim
+ * @param idx A coordDim may map to more than one data dim.
+ * If not specified, return the first dim not extra.
+ * @return concrete data dim. If not found, return null/undefined
+ */
+ mapDimension(coordDim: DimensionName): DimensionName;
+ mapDimension(coordDim: DimensionName, idx: number): DimensionName;
+ mapDimension(coordDim: DimensionName, idx?: number): DimensionName {
+ const dimensionsSummary = this._dimensionsSummary;
+
+ if (idx == null) {
+ return dimensionsSummary.encodeFirstDimNotExtra[coordDim] as any;
+ }
+
+ const dims = dimensionsSummary.encode[coordDim];
+ return dims ? dims[idx as number] as any : null;
+ }
+
+ mapDimensionsAll(coordDim: DimensionName): DimensionName[] {
+ const dimensionsSummary = this._dimensionsSummary;
+ const dims = dimensionsSummary.encode[coordDim];
+ return (dims || []).slice();
+ }
+
+ getStorage() {
+ return this._store;
+ }
+
+ /**
+ * Initialize from data
+ * @param data source or data or data storage.
+ * @param nameList The name of a datum is used on data diff and
+ * default label/tooltip.
+ * A name can be specified in encode.itemName,
+ * or dataItem.name (only for series option data),
+ * or provided in nameList from outside.
+ */
+ initData(
+ data: Source | OptionSourceData | DataStorage | DataProvider,
+ nameList?: string[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ let store: DataStorage;
+ const dimensions = this.dimensions;
+ const dimensionInfos = map(dimensions, dimName => this._dimensionInfos[dimName]);
+ if (data instanceof DataStorage) {
+ if (data.syncDimensionTypes(dimensionInfos)) {
+ store = data;
+ }
+ // Sync failed
+ else {
+ // Needs to recreate data storage if it's not match the given dimension.
+ data = data.getSource();
+ }
+ }
+
+ if (!store) {
+ const provider = (isSourceInstance(data) || zrUtil.isArrayLike(data))
+ ? new DefaultDataProvider(data as Source | OptionSourceData, dimensions.length)
+ : data as DataProvider;
+ store = new DataStorage();
+ store.initData(provider, dimensionInfos, dimValueGetter);
+ }
+
+ this._store = store;
+
+ // Reset
+ this._nameList = (nameList || []).slice();
+ this._idList = [];
+ this._nameRepeatCount = {};
+
+ this._doInit(0, store.count());
+ }
+
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ */
+ appendData(data: ArrayLike<any>): void {
+ const range = this._store.appendData(data);
+ this._doInit(range[0], range[1]);
+ }
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ * This method does not modify `rawData` (`dataProvider`), but only
+ * add values to storage.
+ *
+ * The final count will be increased by `Math.max(values.length, names.length)`.
+ *
+ * @param values That is the SourceType: 'arrayRows', like
+ * [
+ * [12, 33, 44],
+ * [NaN, 43, 1],
+ * ['-', 'asdf', 0]
+ * ]
+ * Each item is exaclty cooresponding to a dimension.
+ */
+ appendValues(values: any[][], names?: string[]): void {
+ const {start, end} = this._store.appendValues(values, names.length);
+ const shouldMakeIdFromName = this._shouldMakeIdFromName();
+
+ this._updateOrdinalMeta();
+
+ if (names) {
+ for (let idx = start; idx < end; idx++) {
+ const sourceIdx = idx - start;
+ this._nameList[idx] = names[sourceIdx];
+ if (shouldMakeIdFromName) {
+ makeIdFromName(this, idx);
+ }
+ }
+ }
+ }
+
+ private _updateOrdinalMeta() {
+ const store = this._store;
+ const dimensions = this.dimensions;
+ for (let i = 0; i < dimensions.length; i++) {
+ const dimInfo = this._dimensionInfos[dimensions[i]];
+ if (dimInfo.ordinalMeta) {
+ const dimIdx = store.getDimensionIndex(dimensions[i]);
+ store.collectOrdinalMeta(dimIdx, dimInfo.ordinalMeta);
+ }
+ }
+ }
+
+ private _shouldMakeIdFromName() {
+ const provider = this._store.getProvider();
+ return this._idDimIdx == null
+ && provider.getSource().sourceFormat !== SOURCE_FORMAT_TYPED_ARRAY
+ && !provider.fillStorage;
+ }
+
+ private _doInit(start: number, end: number): void {
+ if (start >= end) {
+ return;
+ }
+
+ const store = this._store;
+ const provider = store.getProvider();
+
+ this._updateOrdinalMeta();
+
+ const nameList = this._nameList;
+ const idList = this._idList;
+ const sourceFormat = provider.getSource().sourceFormat;
+ const isFormatOriginal = sourceFormat === SOURCE_FORMAT_ORIGINAL;
+
+ // Each data item is value
+ // [1, 2]
+ // 2
+ // Bar chart, line chart which uses category axis
+ // only gives the 'y' value. 'x' value is the indices of category
+ // Use a tempValue to normalize the value to be a (x, y) value
+ // If dataItem is {name: ...} or {id: ...}, it has highest priority.
+ // This kind of ids and names are always stored `_nameList` and `_idList`.
+ if (isFormatOriginal && !provider.pure) {
+ const sharedDataItem = [] as OptionDataItem;
+ for (let idx = start; idx < end; idx++) {
+ // NOTICE: Try not to write things into dataItem
+ const dataItem = provider.getItem(idx, sharedDataItem);
+ if (!this.hasItemOption && isDataItemOption(dataItem)) {
+ this.hasItemOption = true;
+ }
+ if (dataItem) {
+ const itemName = (dataItem as any).name;
+ if (nameList[idx] == null && itemName != null) {
+ nameList[idx] = convertOptionIdName(itemName, null);
+ }
+ const itemId = (dataItem as any).id;
+ if (idList[idx] == null && itemId != null) {
+ idList[idx] = convertOptionIdName(itemId, null);
+ }
+ }
+ }
+ }
+
+ if (this._shouldMakeIdFromName()) {
+ for (let idx = start; idx < end; idx++) {
+ makeIdFromName(this, idx);
+ }
+ }
+
+ prepareInvertedIndex(this);
+ }
+ /**
+ * PENDING: In fact currently this function is only used to short-circuit
+ * the calling of `scale.unionExtentFromData` when data have been filtered by modules
+ * like "dataZoom". `scale.unionExtentFromData` is used to calculate data extent for series on
+ * an axis, but if a "axis related data filter module" is used, the extent of the axis have
+ * been fixed and no need to calling `scale.unionExtentFromData` actually.
+ * But if we add "custom data filter" in future, which is not "axis related", this method may
+ * be still needed.
+ *
+ * Optimize for the scenario that data is filtered by a given extent.
+ * Consider that if data amount is more than hundreds of thousand,
+ * extent calculation will cost more than 10ms and the cache will
+ * be erased because of the filtering.
+ */
+ getApproximateExtent(dim: DimensionLoose): [number, number] {
+ return this._approximateExtent[dim] || this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ /**
+ * Calculate extent on a filtered data might be time consuming.
+ * Approximate extent is only used for: calculte extent of filtered data outside.
+ */
+ setApproximateExtent(extent: [number, number], dim: DimensionLoose): void {
+ dim = this.getDimension(dim);
+ this._approximateExtent[dim] = extent.slice() as [number, number];
+ }
+
+ getCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY
+ ): DataCalculationInfo<HostModel>[CALC_INFO_KEY] {
+ return this._calculationInfo[key];
+ }
+
+ /**
+ * @param key or k-v object
+ */
+ setCalculationInfo(
+ key: DataCalculationInfo<HostModel>
+ ): void;
+ setCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY,
+ value: DataCalculationInfo<HostModel>[CALC_INFO_KEY]
+ ): void;
+ setCalculationInfo(
+ key: (keyof DataCalculationInfo<HostModel>) | DataCalculationInfo<HostModel>,
+ value?: DataCalculationInfo<HostModel>[keyof DataCalculationInfo<HostModel>]
+ ): void {
+ isObject(key)
+ ? zrUtil.extend(this._calculationInfo, key as object)
+ : ((this._calculationInfo as any)[key] = value);
+ }
+
+ /**
+ * @return Never be null/undefined. `number` will be converted to string. Becuase:
+ * In most cases, name is used in display, where returning a string is more convenient.
+ * In other cases, name is used in query (see `indexOfName`), where we can keep the
+ * rule that name `2` equals to name `'2'`.
+ */
+ getName(idx: number): string {
+ const rawIndex = this.getRawIndex(idx);
+ let name = this._nameList[rawIndex];
+ if (name == null && this._nameDimIdx != null) {
+ name = getIdNameFromStore(this, this._nameDimIdx, rawIndex);
+ }
+ if (name == null) {
+ name = '';
+ }
+ return name;
+ }
+
+ private _getCategory(dimIdx: number, idx: number) {
+ const ordinal = this._store.get(dimIdx, idx);
+ const ordinalMeta = this._store.getOrdinalMeta(dimIdx);
+ if (ordinalMeta) {
+ return ordinalMeta.categories[ordinal as OrdinalNumber];
+ }
+ return ordinal;
+ }
+
+ /**
+ * @return Never null/undefined. `number` will be converted to string. Becuase:
+ * In all cases having encountered at present, id is used in making diff comparison, which
+ * are usually based on hash map. We can keep the rule that the internal id are always string
+ * (treat `2` is the same as `'2'`) to make the related logic simple.
+ */
+ getId(idx: number): string {
+ return getId(this, this.getRawIndex(idx));
+ }
+
+ count() {
+ return this._store.count();
+ }
+ /**
+ * Get value. Return NaN if idx is out of range.
+ * @param dim Dim must be concrete name.
+ */
+ get(dim: DimensionName, idx: number): ParsedValue {
+ const store = this._store;
+ return store.get(store.getDimensionIndex(dim), idx);
+ }
+
+ getByRawIndex(dim: DimensionName, rawIdx: number): ParsedValue {
+ const store = this._store;
+ return store.getByRawIndex(store.getDimensionIndex(dim), rawIdx);
+ }
+
+ getIndices() {
+ return this._store.getIndices();
+ }
+
+ getDataExtent(dim: DimensionLoose): [number, number] {
+ return this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ getSum(dim: DimensionLoose): number {
+ return this._store.getSum(this._getStoreDimIndex(dim));
+ }
+
+ getMedian(dim: DimensionLoose): number {
+ return this._store.getMedian(this._getStoreDimIndex(dim));
+ }
+ /**
+ * Get value for multi dimensions.
+ * @param dimensions If ignored, using all dimensions.
+ */
+ getValues(idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[], idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[] | number, idx?: number): ParsedValue[] {
+ const store = this._store;
+ return zrUtil.isArray(dimensions)
+ ? store.getValues(map(dimensions, dim => this._getStoreDimIndex(dim)), idx)
+ : store.getValues(dimensions as number);
+ }
+
+ /**
+ * If value is NaN. Inlcuding '-'
+ * Only check the coord dimensions.
+ */
+ hasValue(idx: number): boolean {
+ const dataDimsOnCoord = this._dimensionsSummary.dataDimsOnCoord;
+ for (let i = 0, len = dataDimsOnCoord.length; i < len; i++) {
+ // Ordinal type originally can be string or number.
+ // But when an ordinal type is used on coord, it can
+ // not be string but only number. So we can also use isNaN.
+ if (isNaN(this.get(dataDimsOnCoord[i], idx) as any)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Retreive the index with given name
+ */
+ indexOfName(name: string): number {
+ for (let i = 0, len = this._store.count(); i < len; i++) {
+ if (this.getName(i) === name) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ getRawIndex(idx: number) {
+ return this._store.getRawIndex(idx);
+ }
+
+ indexOfRawIndex(rawIndex: number) {
+ return this._store.indexOfRawIndex(rawIndex);
+ }
+
+ /**
+ * Only support the dimension which inverted index created.
+ * Do not support other cases until required.
+ * @param dim concrete dim
+ * @param value ordinal index
+ * @return rawIndex
+ */
+ rawIndexOf(dim: DimensionName, value: OrdinalNumber): number {
+ const invertedIndices = dim && this._invertedIndicesMap[dim];
+ if (__DEV__) {
+ if (!invertedIndices) {
+ throw new Error('Do not supported yet');
+ }
+ }
+ const rawIndex = invertedIndices[value];
+ if (rawIndex == null || isNaN(rawIndex)) {
+ return INDEX_NOT_FOUND;
+ }
+ return rawIndex;
+ }
+
+ /**
+ * Retreive the index of nearest value
+ * @param dim
+ * @param value
+ * @param [maxDistance=Infinity]
+ * @return If and only if multiple indices has
+ * the same value, they are put to the result.
+ */
+ indicesOfNearest(dim: DimensionLoose, value: number, maxDistance?: number): number[] {
+ return this._store.indicesOfNearest(
+ this._getStoreDimIndex(dim),
+ value, maxDistance
+ );
+ }
+ /**
+ * Data iteration
+ * @param ctx default this
+ * @example
+ * list.each('x', function (x, idx) {});
+ * list.each(['x', 'y'], function (x, y, idx) {});
+ * list.each(function (idx) {})
+ */
+ each<Ctx>(cb: EachCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): void;
+ each<Ctx>(dims: DimensionLoose, cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose], cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: EachCb2<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: ItrParamDims, cb: EachCb<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(
+ dims: ItrParamDims | EachCb<Ctx>,
+ cb: EachCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): void {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store.each(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ }
+ /**
+ * Data filter
+ */
+ filterSelf<Ctx>(cb: FilterCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): this;
+ filterSelf<Ctx>(dims: DimensionLoose, cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose], cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: FilterCb2<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: ItrParamDims, cb: FilterCb<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(
+ dims: ItrParamDims | FilterCb<Ctx>,
+ cb: FilterCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ // Clone first
+ this._store = this._store.clone();
+ this._store.filterSelf(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+
+ return this;
+ }
+
+ /**
+ * Select data in range. (For optimization of filter)
+ * (Manually inline code, support 5 million data filtering in data zoom.)
+ */
+ selectRange(range: Record<string, [number, number]>): SeriesData {
+ 'use strict';
+
+ const innerRange: Record<number, [number, number]> = {};
+ const dims = zrUtil.keys(range);
+ const dimIndices: number[] = [];
+ zrUtil.each(dims, (dim) => {
+ const dimIdx = this._getStoreDimIndex(dim);
+ innerRange[dimIdx] = range[dim];
+ dimIndices.push(dimIdx);
+ });
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store = this._store.clone();
+ this._store.selectRange(innerRange);
+ return this;
+ }
+
+ /**
+ * Data mapping to a plain array
+ */
+ mapArray<Ctx, Cb extends MapArrayCb0<Ctx>>(cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-disable */
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: DimensionLoose, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: [DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb2<Ctx>>(dims: [DimensionLoose, DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb<Ctx>>(dims: ItrParamDims, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-enable */
+ mapArray<Ctx>(
+ dims: ItrParamDims | MapArrayCb<Ctx>,
+ cb: MapArrayCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): unknown[] {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ ctx = (ctx || this) as Ctx;
+
+ const result: unknown[] = [];
+ this.each(dims, function () {
+ result.push(cb && (cb as MapArrayCb<Ctx>).apply(this, arguments));
+ }, ctx);
+ return result;
+ }
+
+ /**
+ * Data mapping to a new List with given dimensions
+ */
+ map<Ctx>(dims: DimensionLoose, cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(dims: [DimensionLoose], cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ /* eslint-disable-next-line */
+ map<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: MapCb2<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(
+ dims: ItrParamDims,
+ cb: MapCb<Ctx>,
+ ctx?: Ctx,
+ ctxCompat?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || ctxCompat || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(
+ normalizeDimensions(dims), this._getStoreDimIndex, this
+ );
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ const list = cloneListForMapAndSample(this);
+ list._store = this._store.map(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ return list;
+ }
+
+ /**
+ * Large data down sampling on given dimension
+ * @param sampleIndex Sample index for name and id
+ */
+ downSample(
Review comment:
Yes I agree, `DataStorage` will be totally immutable. (The only exception is `appendData` and `appendValues`, which are restricted to only be called in `Series#getRawData`).
##########
File path: src/data/SeriesData.ts
##########
@@ -0,0 +1,1338 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+/**
+ * List for data storage
+ */
+
+import * as zrUtil from 'zrender/src/core/util';
+import {PathStyleProps} from 'zrender/src/graphic/Path';
+import Model from '../model/Model';
+import DataDiffer from './DataDiffer';
+import {DataProvider, DefaultDataProvider} from './helper/dataProvider';
+import {summarizeDimensions, DimensionSummary} from './helper/dimensionHelper';
+import DataDimensionInfo from './DataDimensionInfo';
+import {ArrayLike, Dictionary, FunctionPropertyNames} from 'zrender/src/core/types';
+import Element from 'zrender/src/Element';
+import {
+ DimensionIndex, DimensionName, DimensionLoose, OptionDataItem,
+ ParsedValue, ParsedValueNumeric, DimensionUserOuput,
+ ModelOption, SeriesDataType, OptionSourceData, SOURCE_FORMAT_TYPED_ARRAY, SOURCE_FORMAT_ORIGINAL,
+ DecalObject,
+ OrdinalNumber
+} from '../util/types';
+import {convertOptionIdName, isDataItemOption} from '../util/model';
+import { setCommonECData } from '../util/innerStore';
+import type Graph from './Graph';
+import type Tree from './Tree';
+import type { VisualMeta } from '../component/visualMap/VisualMapModel';
+import {isSourceInstance, Source} from './Source';
+import { LineStyleProps } from '../model/mixin/lineStyle';
+import DataStorage, { DimValueGetter } from './DataStorage';
+
+const isObject = zrUtil.isObject;
+const map = zrUtil.map;
+
+const CtorInt32Array = typeof Int32Array === 'undefined' ? Array : Int32Array;
+
+// Use prefix to avoid index to be the same as otherIdList[idx],
+// which will cause weird udpate animation.
+const ID_PREFIX = 'e\0\0';
+
+const INDEX_NOT_FOUND = -1;
+
+type NameRepeatCount = {[name: string]: number};
+type ItrParamDims = DimensionLoose | Array<DimensionLoose>;
+// If Ctx not specified, use List as Ctx
+type CtxOrList<Ctx> = unknown extends Ctx ? SeriesData : Ctx;
+type EachCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => void;
+type EachCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => void;
+type EachCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => void;
+type FilterCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => boolean;
+type FilterCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => boolean;
+type FilterCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => boolean;
+type FilterCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => boolean;
+type MapArrayCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => any;
+type MapArrayCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => any;
+type MapArrayCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => any;
+type MapArrayCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => any;
+type MapCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => ParsedValue | ParsedValue[];
+type MapCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) =>
+ ParsedValue | ParsedValue[];
+type MapCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => ParsedValue | ParsedValue[];
+
+const TRANSFERABLE_PROPERTIES = [
+ 'hasItemOption', '_nameList', '_idList', '_invertedIndicesMap',
+ '_rawData', '_dimValueGetter',
+ '_nameDimIdx', '_idDimIdx', '_nameRepeatCount'
+];
+const CLONE_PROPERTIES = [
+ '_approximateExtent'
+];
+
+export interface DefaultDataVisual {
+ style: PathStyleProps
+ // Draw type determined which prop should be set with encoded color.
+ // It's only available on the global visual. Use getVisual('drawType') to access it.
+ // It will be set in visual/style.ts module in the first priority.
+ drawType: 'fill' | 'stroke'
+
+ symbol?: string
+ symbolSize?: number | number[]
+ symbolRotate?: number
+ symbolKeepAspect?: boolean
+ symbolOffset?: string | number | (string | number)[]
+
+ liftZ?: number
+ // For legend.
+ legendIcon?: string
+ legendLineStyle?: LineStyleProps
+
+ // visualMap will inject visualMeta data
+ visualMeta?: VisualMeta[]
+
+ // If color is encoded from palette
+ colorFromPalette?: boolean
+
+ decal?: DecalObject
+}
+
+export interface DataCalculationInfo<SERIES_MODEL> {
+ stackedDimension: string;
+ stackedByDimension: string;
+ isStackedByIndex: boolean;
+ stackedOverDimension: string;
+ stackResultDimension: string;
+ stackedOnSeries?: SERIES_MODEL;
+}
+
+// -----------------------------
+// Internal method declarations:
+// -----------------------------
+let prepareInvertedIndex: (data: SeriesData) => void;
+let getId: (data: SeriesData, rawIndex: number) => string;
+let getIdNameFromStore: (data: SeriesData, dimIdx: number, dataIdx: number) => string;
+let normalizeDimensions: (dimensions: ItrParamDims) => Array<DimensionLoose>;
+let validateDimensions: (data: SeriesData, dims: DimensionIndex[]) => void;
+let transferProperties: (target: SeriesData, source: SeriesData) => void;
+let cloneListForMapAndSample: (original: SeriesData) => SeriesData;
+let makeIdFromName: (data: SeriesData, idx: number) => void;
+
+class SeriesData<
+ HostModel extends Model = Model,
+ Visual extends DefaultDataVisual = DefaultDataVisual
+> {
+
+ readonly type = 'list';
+
+ readonly dimensions: string[];
+
+ // Infomation of each data dimension, like data type.
+ private _dimensionInfos: {[dimName: string]: DataDimensionInfo};
+
+ readonly hostModel: HostModel;
+
+ /**
+ * @readonly
+ */
+ dataType: SeriesDataType;
+
+ /**
+ * @readonly
+ * Host graph if List is used to store graph nodes / edges.
+ */
+ graph?: Graph;
+
+ /**
+ * @readonly
+ * Host tree if List is used to store tree ndoes.
+ */
+ tree?: Tree;
+
+ private _store: DataStorage;
+
+ private _nameList: string[] = [];
+ private _idList: string[] = [];
+
+ // Models of data option is stored sparse for optimizing memory cost
+ // Never used yet (not used yet).
+ // private _optionModels: Model[] = [];
+
+ // Global visual properties after visual coding
+ private _visual: Dictionary<any> = {};
+
+ // Globel layout properties.
+ private _layout: Dictionary<any> = {};
+
+ // Item visual properties after visual coding
+ private _itemVisuals: Dictionary<any>[] = [];
+
+ // Item layout properties after layout
+ private _itemLayouts: any[] = [];
+
+ // Graphic elemnents
+ private _graphicEls: Element[] = [];
+
+ // key: dim, value: extent
+ private _approximateExtent: {[dimName: string]: [number, number]} = {};
+
+ private _dimensionsSummary: DimensionSummary;
+
+ private _invertedIndicesMap: {[dimName: string]: ArrayLike<number>};
+
+ private _calculationInfo: DataCalculationInfo<HostModel> = {} as DataCalculationInfo<HostModel>;
+
+ // User output info of this data.
+ // DO NOT use it in other places!
+ // When preparing user params for user callbacks, we have
+ // to clone these inner data structures to prevent users
+ // from modifying them to effect built-in logic. And for
+ // performance consideration we make this `userOutput` to
+ // avoid clone them too many times.
+ readonly userOutput: DimensionUserOuput;
+
+ // Having detected that there is data item is non primitive type
+ // (in type `OptionDataItemObject`).
+ // Like `data: [ { value: xx, itemStyle: {...} }, ...]`
+ // At present it only happen in `SOURCE_FORMAT_ORIGINAL`.
+ hasItemOption: boolean = false;
+
+ // id or name is used on dynamic data, mapping old and new items.
+ // When generating id from name, avoid repeat.
+ private _nameRepeatCount: NameRepeatCount;
+ private _nameDimIdx: number;
+ private _idDimIdx: number;
+
+ private __wrappedMethods: string[];
+
+ // Methods that create a new list based on this list should be listed here.
+ // Notice that those method should `RETURN` the new list.
+ TRANSFERABLE_METHODS = ['cloneShallow', 'downSample', 'lttbDownSample', 'map'] as const;
+ // Methods that change indices of this list should be listed here.
+ CHANGABLE_METHODS = ['filterSelf', 'selectRange'] as const;
+ DOWNSAMPLE_METHODS = ['downSample', 'lttbDownSample'] as const;
+
+ /**
+ * @param dimensions
+ * For example, ['someDimName', {name: 'someDimName', type: 'someDimType'}, ...].
+ * Dimensions should be concrete names like x, y, z, lng, lat, angle, radius
+ */
+ constructor(dimensions: Array<string | object | DataDimensionInfo>, hostModel: HostModel) {
+ dimensions = dimensions || ['x', 'y'];
+
+ const dimensionInfos: Dictionary<DataDimensionInfo> = {};
+ const dimensionNames = [];
+ const invertedIndicesMap: Dictionary<number[]> = {};
+
+ for (let i = 0; i < dimensions.length; i++) {
+ // Use the original dimensions[i], where other flag props may exists.
+ const dimInfoInput = dimensions[i];
+
+ const dimensionInfo: DataDimensionInfo =
+ zrUtil.isString(dimInfoInput)
+ ? new DataDimensionInfo({name: dimInfoInput})
+ : !(dimInfoInput instanceof DataDimensionInfo)
+ ? new DataDimensionInfo(dimInfoInput)
+ : dimInfoInput;
+
+ const dimensionName = dimensionInfo.name;
+ dimensionInfo.type = dimensionInfo.type || 'float';
+ if (!dimensionInfo.coordDim) {
+ dimensionInfo.coordDim = dimensionName;
+ dimensionInfo.coordDimIndex = 0;
+ }
+
+ const otherDims = dimensionInfo.otherDims = dimensionInfo.otherDims || {};
+ dimensionNames.push(dimensionName);
+ dimensionInfos[dimensionName] = dimensionInfo;
+
+ dimensionInfo.index = i;
+
+ if (dimensionInfo.createInvertedIndices) {
+ invertedIndicesMap[dimensionName] = [];
+ }
+ if (otherDims.itemName === 0) {
+ this._nameDimIdx = i;
+ }
+ if (otherDims.itemId === 0) {
+ this._idDimIdx = i;
+ }
+ }
+
+ this.dimensions = dimensionNames;
+ this._dimensionInfos = dimensionInfos;
+ this.hostModel = hostModel;
+
+ // Cache summary info for fast visit. See "dimensionHelper".
+ this._dimensionsSummary = summarizeDimensions(this);
+
+ this._invertedIndicesMap = invertedIndicesMap;
+
+ this.userOutput = this._dimensionsSummary.userOutput;
+ }
+
+ /**
+ * The meanings of the input parameter `dim`:
+ *
+ * + If dim is a number (e.g., `1`), it means the index of the dimension.
+ * For example, `getDimension(0)` will return 'x' or 'lng' or 'radius'.
+ * + If dim is a number-like string (e.g., `"1"`):
+ * + If there is the same concrete dim name defined in `this.dimensions`, it means that concrete name.
+ * + If not, it will be converted to a number, which means the index of the dimension.
+ * (why? because of the backward compatbility. We have been tolerating number-like string in
+ * dimension setting, although now it seems that it is not a good idea.)
+ * For example, `visualMap[i].dimension: "1"` is the same meaning as `visualMap[i].dimension: 1`,
+ * if no dimension name is defined as `"1"`.
+ * + If dim is a not-number-like string, it means the concrete dim name.
+ * For example, it can be be default name `"x"`, `"y"`, `"z"`, `"lng"`, `"lat"`, `"angle"`, `"radius"`,
+ * or customized in `dimensions` property of option like `"age"`.
+ *
+ * Get dimension name
+ * @param dim See above.
+ * @return Concrete dim name.
+ */
+ getDimension(dim: DimensionLoose): DimensionName {
+ if (typeof dim === 'number'
+ // If being a number-like string but not being defined a dimension name.
+ || (!isNaN(dim as any) && !this._dimensionInfos.hasOwnProperty(dim))
+ ) {
+ dim = this.dimensions[dim as DimensionIndex];
+ }
+ return dim as DimensionName;
+ }
+
+ private _getStoreDimIndex(dim: DimensionLoose): DimensionIndex {
+ return this._store.getDimensionIndex(this.getDimension(dim));
+ }
+
+ /**
+ * Get type and calculation info of particular dimension
+ * @param dim
+ * Dimension can be concrete names like x, y, z, lng, lat, angle, radius
+ * Or a ordinal number. For example getDimensionInfo(0) will return 'x' or 'lng' or 'radius'
+ */
+ getDimensionInfo(dim: DimensionLoose): DataDimensionInfo {
+ // Do not clone, because there may be categories in dimInfo.
+ return this._dimensionInfos[this.getDimension(dim)];
+ }
+
+ /**
+ * concrete dimension name list on coord.
+ */
+ getDimensionsOnCoord(): DimensionName[] {
+ return this._dimensionsSummary.dataDimsOnCoord.slice();
+ }
+
+ /**
+ * @param coordDim
+ * @param idx A coordDim may map to more than one data dim.
+ * If not specified, return the first dim not extra.
+ * @return concrete data dim. If not found, return null/undefined
+ */
+ mapDimension(coordDim: DimensionName): DimensionName;
+ mapDimension(coordDim: DimensionName, idx: number): DimensionName;
+ mapDimension(coordDim: DimensionName, idx?: number): DimensionName {
+ const dimensionsSummary = this._dimensionsSummary;
+
+ if (idx == null) {
+ return dimensionsSummary.encodeFirstDimNotExtra[coordDim] as any;
+ }
+
+ const dims = dimensionsSummary.encode[coordDim];
+ return dims ? dims[idx as number] as any : null;
+ }
+
+ mapDimensionsAll(coordDim: DimensionName): DimensionName[] {
+ const dimensionsSummary = this._dimensionsSummary;
+ const dims = dimensionsSummary.encode[coordDim];
+ return (dims || []).slice();
+ }
+
+ getStorage() {
+ return this._store;
+ }
+
+ /**
+ * Initialize from data
+ * @param data source or data or data storage.
+ * @param nameList The name of a datum is used on data diff and
+ * default label/tooltip.
+ * A name can be specified in encode.itemName,
+ * or dataItem.name (only for series option data),
+ * or provided in nameList from outside.
+ */
+ initData(
+ data: Source | OptionSourceData | DataStorage | DataProvider,
+ nameList?: string[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ let store: DataStorage;
+ const dimensions = this.dimensions;
+ const dimensionInfos = map(dimensions, dimName => this._dimensionInfos[dimName]);
+ if (data instanceof DataStorage) {
+ if (data.syncDimensionTypes(dimensionInfos)) {
+ store = data;
+ }
+ // Sync failed
+ else {
+ // Needs to recreate data storage if it's not match the given dimension.
+ data = data.getSource();
+ }
+ }
+
+ if (!store) {
+ const provider = (isSourceInstance(data) || zrUtil.isArrayLike(data))
+ ? new DefaultDataProvider(data as Source | OptionSourceData, dimensions.length)
+ : data as DataProvider;
+ store = new DataStorage();
+ store.initData(provider, dimensionInfos, dimValueGetter);
+ }
+
+ this._store = store;
+
+ // Reset
+ this._nameList = (nameList || []).slice();
+ this._idList = [];
+ this._nameRepeatCount = {};
+
+ this._doInit(0, store.count());
+ }
+
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ */
+ appendData(data: ArrayLike<any>): void {
+ const range = this._store.appendData(data);
+ this._doInit(range[0], range[1]);
+ }
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ * This method does not modify `rawData` (`dataProvider`), but only
+ * add values to storage.
+ *
+ * The final count will be increased by `Math.max(values.length, names.length)`.
+ *
+ * @param values That is the SourceType: 'arrayRows', like
+ * [
+ * [12, 33, 44],
+ * [NaN, 43, 1],
+ * ['-', 'asdf', 0]
+ * ]
+ * Each item is exaclty cooresponding to a dimension.
+ */
+ appendValues(values: any[][], names?: string[]): void {
+ const {start, end} = this._store.appendValues(values, names.length);
+ const shouldMakeIdFromName = this._shouldMakeIdFromName();
+
+ this._updateOrdinalMeta();
+
+ if (names) {
+ for (let idx = start; idx < end; idx++) {
+ const sourceIdx = idx - start;
+ this._nameList[idx] = names[sourceIdx];
+ if (shouldMakeIdFromName) {
+ makeIdFromName(this, idx);
+ }
+ }
+ }
+ }
+
+ private _updateOrdinalMeta() {
+ const store = this._store;
+ const dimensions = this.dimensions;
+ for (let i = 0; i < dimensions.length; i++) {
+ const dimInfo = this._dimensionInfos[dimensions[i]];
+ if (dimInfo.ordinalMeta) {
+ const dimIdx = store.getDimensionIndex(dimensions[i]);
+ store.collectOrdinalMeta(dimIdx, dimInfo.ordinalMeta);
+ }
+ }
+ }
+
+ private _shouldMakeIdFromName() {
+ const provider = this._store.getProvider();
+ return this._idDimIdx == null
+ && provider.getSource().sourceFormat !== SOURCE_FORMAT_TYPED_ARRAY
+ && !provider.fillStorage;
+ }
+
+ private _doInit(start: number, end: number): void {
+ if (start >= end) {
+ return;
+ }
+
+ const store = this._store;
+ const provider = store.getProvider();
+
+ this._updateOrdinalMeta();
+
+ const nameList = this._nameList;
+ const idList = this._idList;
+ const sourceFormat = provider.getSource().sourceFormat;
+ const isFormatOriginal = sourceFormat === SOURCE_FORMAT_ORIGINAL;
+
+ // Each data item is value
+ // [1, 2]
+ // 2
+ // Bar chart, line chart which uses category axis
+ // only gives the 'y' value. 'x' value is the indices of category
+ // Use a tempValue to normalize the value to be a (x, y) value
+ // If dataItem is {name: ...} or {id: ...}, it has highest priority.
+ // This kind of ids and names are always stored `_nameList` and `_idList`.
+ if (isFormatOriginal && !provider.pure) {
+ const sharedDataItem = [] as OptionDataItem;
+ for (let idx = start; idx < end; idx++) {
+ // NOTICE: Try not to write things into dataItem
+ const dataItem = provider.getItem(idx, sharedDataItem);
+ if (!this.hasItemOption && isDataItemOption(dataItem)) {
+ this.hasItemOption = true;
+ }
+ if (dataItem) {
+ const itemName = (dataItem as any).name;
+ if (nameList[idx] == null && itemName != null) {
+ nameList[idx] = convertOptionIdName(itemName, null);
+ }
+ const itemId = (dataItem as any).id;
+ if (idList[idx] == null && itemId != null) {
+ idList[idx] = convertOptionIdName(itemId, null);
+ }
+ }
+ }
+ }
+
+ if (this._shouldMakeIdFromName()) {
+ for (let idx = start; idx < end; idx++) {
+ makeIdFromName(this, idx);
+ }
+ }
+
+ prepareInvertedIndex(this);
+ }
+ /**
+ * PENDING: In fact currently this function is only used to short-circuit
+ * the calling of `scale.unionExtentFromData` when data have been filtered by modules
+ * like "dataZoom". `scale.unionExtentFromData` is used to calculate data extent for series on
+ * an axis, but if a "axis related data filter module" is used, the extent of the axis have
+ * been fixed and no need to calling `scale.unionExtentFromData` actually.
+ * But if we add "custom data filter" in future, which is not "axis related", this method may
+ * be still needed.
+ *
+ * Optimize for the scenario that data is filtered by a given extent.
+ * Consider that if data amount is more than hundreds of thousand,
+ * extent calculation will cost more than 10ms and the cache will
+ * be erased because of the filtering.
+ */
+ getApproximateExtent(dim: DimensionLoose): [number, number] {
+ return this._approximateExtent[dim] || this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ /**
+ * Calculate extent on a filtered data might be time consuming.
+ * Approximate extent is only used for: calculte extent of filtered data outside.
+ */
+ setApproximateExtent(extent: [number, number], dim: DimensionLoose): void {
+ dim = this.getDimension(dim);
+ this._approximateExtent[dim] = extent.slice() as [number, number];
+ }
+
+ getCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY
+ ): DataCalculationInfo<HostModel>[CALC_INFO_KEY] {
+ return this._calculationInfo[key];
+ }
+
+ /**
+ * @param key or k-v object
+ */
+ setCalculationInfo(
+ key: DataCalculationInfo<HostModel>
+ ): void;
+ setCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY,
+ value: DataCalculationInfo<HostModel>[CALC_INFO_KEY]
+ ): void;
+ setCalculationInfo(
+ key: (keyof DataCalculationInfo<HostModel>) | DataCalculationInfo<HostModel>,
+ value?: DataCalculationInfo<HostModel>[keyof DataCalculationInfo<HostModel>]
+ ): void {
+ isObject(key)
+ ? zrUtil.extend(this._calculationInfo, key as object)
+ : ((this._calculationInfo as any)[key] = value);
+ }
+
+ /**
+ * @return Never be null/undefined. `number` will be converted to string. Becuase:
+ * In most cases, name is used in display, where returning a string is more convenient.
+ * In other cases, name is used in query (see `indexOfName`), where we can keep the
+ * rule that name `2` equals to name `'2'`.
+ */
+ getName(idx: number): string {
+ const rawIndex = this.getRawIndex(idx);
+ let name = this._nameList[rawIndex];
+ if (name == null && this._nameDimIdx != null) {
+ name = getIdNameFromStore(this, this._nameDimIdx, rawIndex);
+ }
+ if (name == null) {
+ name = '';
+ }
+ return name;
+ }
+
+ private _getCategory(dimIdx: number, idx: number) {
+ const ordinal = this._store.get(dimIdx, idx);
+ const ordinalMeta = this._store.getOrdinalMeta(dimIdx);
+ if (ordinalMeta) {
+ return ordinalMeta.categories[ordinal as OrdinalNumber];
+ }
+ return ordinal;
+ }
+
+ /**
+ * @return Never null/undefined. `number` will be converted to string. Becuase:
+ * In all cases having encountered at present, id is used in making diff comparison, which
+ * are usually based on hash map. We can keep the rule that the internal id are always string
+ * (treat `2` is the same as `'2'`) to make the related logic simple.
+ */
+ getId(idx: number): string {
+ return getId(this, this.getRawIndex(idx));
+ }
+
+ count() {
+ return this._store.count();
+ }
+ /**
+ * Get value. Return NaN if idx is out of range.
+ * @param dim Dim must be concrete name.
+ */
+ get(dim: DimensionName, idx: number): ParsedValue {
+ const store = this._store;
+ return store.get(store.getDimensionIndex(dim), idx);
+ }
+
+ getByRawIndex(dim: DimensionName, rawIdx: number): ParsedValue {
+ const store = this._store;
+ return store.getByRawIndex(store.getDimensionIndex(dim), rawIdx);
+ }
+
+ getIndices() {
+ return this._store.getIndices();
+ }
+
+ getDataExtent(dim: DimensionLoose): [number, number] {
+ return this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ getSum(dim: DimensionLoose): number {
+ return this._store.getSum(this._getStoreDimIndex(dim));
+ }
+
+ getMedian(dim: DimensionLoose): number {
+ return this._store.getMedian(this._getStoreDimIndex(dim));
+ }
+ /**
+ * Get value for multi dimensions.
+ * @param dimensions If ignored, using all dimensions.
+ */
+ getValues(idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[], idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[] | number, idx?: number): ParsedValue[] {
+ const store = this._store;
+ return zrUtil.isArray(dimensions)
+ ? store.getValues(map(dimensions, dim => this._getStoreDimIndex(dim)), idx)
+ : store.getValues(dimensions as number);
+ }
+
+ /**
+ * If value is NaN. Inlcuding '-'
+ * Only check the coord dimensions.
+ */
+ hasValue(idx: number): boolean {
+ const dataDimsOnCoord = this._dimensionsSummary.dataDimsOnCoord;
+ for (let i = 0, len = dataDimsOnCoord.length; i < len; i++) {
+ // Ordinal type originally can be string or number.
+ // But when an ordinal type is used on coord, it can
+ // not be string but only number. So we can also use isNaN.
+ if (isNaN(this.get(dataDimsOnCoord[i], idx) as any)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Retreive the index with given name
+ */
+ indexOfName(name: string): number {
+ for (let i = 0, len = this._store.count(); i < len; i++) {
+ if (this.getName(i) === name) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ getRawIndex(idx: number) {
+ return this._store.getRawIndex(idx);
+ }
+
+ indexOfRawIndex(rawIndex: number) {
+ return this._store.indexOfRawIndex(rawIndex);
+ }
+
+ /**
+ * Only support the dimension which inverted index created.
+ * Do not support other cases until required.
+ * @param dim concrete dim
+ * @param value ordinal index
+ * @return rawIndex
+ */
+ rawIndexOf(dim: DimensionName, value: OrdinalNumber): number {
+ const invertedIndices = dim && this._invertedIndicesMap[dim];
+ if (__DEV__) {
+ if (!invertedIndices) {
+ throw new Error('Do not supported yet');
+ }
+ }
+ const rawIndex = invertedIndices[value];
+ if (rawIndex == null || isNaN(rawIndex)) {
+ return INDEX_NOT_FOUND;
+ }
+ return rawIndex;
+ }
+
+ /**
+ * Retreive the index of nearest value
+ * @param dim
+ * @param value
+ * @param [maxDistance=Infinity]
+ * @return If and only if multiple indices has
+ * the same value, they are put to the result.
+ */
+ indicesOfNearest(dim: DimensionLoose, value: number, maxDistance?: number): number[] {
+ return this._store.indicesOfNearest(
+ this._getStoreDimIndex(dim),
+ value, maxDistance
+ );
+ }
+ /**
+ * Data iteration
+ * @param ctx default this
+ * @example
+ * list.each('x', function (x, idx) {});
+ * list.each(['x', 'y'], function (x, y, idx) {});
+ * list.each(function (idx) {})
+ */
+ each<Ctx>(cb: EachCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): void;
+ each<Ctx>(dims: DimensionLoose, cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose], cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: EachCb2<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: ItrParamDims, cb: EachCb<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(
+ dims: ItrParamDims | EachCb<Ctx>,
+ cb: EachCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): void {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store.each(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ }
+ /**
+ * Data filter
+ */
+ filterSelf<Ctx>(cb: FilterCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): this;
+ filterSelf<Ctx>(dims: DimensionLoose, cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose], cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: FilterCb2<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: ItrParamDims, cb: FilterCb<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(
+ dims: ItrParamDims | FilterCb<Ctx>,
+ cb: FilterCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ // Clone first
+ this._store = this._store.clone();
+ this._store.filterSelf(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+
+ return this;
+ }
+
+ /**
+ * Select data in range. (For optimization of filter)
+ * (Manually inline code, support 5 million data filtering in data zoom.)
+ */
+ selectRange(range: Record<string, [number, number]>): SeriesData {
+ 'use strict';
+
+ const innerRange: Record<number, [number, number]> = {};
+ const dims = zrUtil.keys(range);
+ const dimIndices: number[] = [];
+ zrUtil.each(dims, (dim) => {
+ const dimIdx = this._getStoreDimIndex(dim);
+ innerRange[dimIdx] = range[dim];
+ dimIndices.push(dimIdx);
+ });
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store = this._store.clone();
+ this._store.selectRange(innerRange);
+ return this;
+ }
+
+ /**
+ * Data mapping to a plain array
+ */
+ mapArray<Ctx, Cb extends MapArrayCb0<Ctx>>(cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-disable */
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: DimensionLoose, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: [DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb2<Ctx>>(dims: [DimensionLoose, DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb<Ctx>>(dims: ItrParamDims, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-enable */
+ mapArray<Ctx>(
+ dims: ItrParamDims | MapArrayCb<Ctx>,
+ cb: MapArrayCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): unknown[] {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ ctx = (ctx || this) as Ctx;
+
+ const result: unknown[] = [];
+ this.each(dims, function () {
+ result.push(cb && (cb as MapArrayCb<Ctx>).apply(this, arguments));
+ }, ctx);
+ return result;
+ }
+
+ /**
+ * Data mapping to a new List with given dimensions
+ */
+ map<Ctx>(dims: DimensionLoose, cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(dims: [DimensionLoose], cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ /* eslint-disable-next-line */
+ map<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: MapCb2<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(
+ dims: ItrParamDims,
+ cb: MapCb<Ctx>,
+ ctx?: Ctx,
+ ctxCompat?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || ctxCompat || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(
+ normalizeDimensions(dims), this._getStoreDimIndex, this
+ );
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ const list = cloneListForMapAndSample(this);
+ list._store = this._store.map(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ return list;
+ }
+
+ /**
+ * Large data down sampling on given dimension
+ * @param sampleIndex Sample index for name and id
+ */
+ downSample(
Review comment:
Yes I agree, `DataStorage` should be totally immutable. (The only exception is `appendData` and `appendValues`, which are restricted to only be called in `Series#getRawData`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675146798
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
> We should support user-defined 'null' / 'undefined' to be parsed to null / undefined?
1. I mean add `userDimName != null` before `dataDimNameMap.get(userDimName)`.
2. I mean might be
```ts
if (userDimName != null && dataDimNameMap.get(userDimName) == null) { // change != to ==
resultItem.name = resultItem.displayName = userDimName;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675303987
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
I split the original logic in https://github.com/apache/echarts/blob/master/src/data/helper/completeDimensions.ts#L115 into two:
https://github.com/apache/echarts/blob/3de73abe262912b928e23b6eab6ac568a67bd7d7/src/data/helper/createDimensions.ts#L156
and
https://github.com/apache/echarts/blob/3de73abe262912b928e23b6eab6ac568a67bd7d7/src/data/helper/createDimensions.ts#L139
Because I need to create and set the result dimension only when it's used.
But seems the logic is not totally equal and may have some issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And always use `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like (1) dimensionDefineList, (2) whether dimension omitted, (3) source for this series, (4) ...).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source (only when "dimension omit" happen). If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
## About `SeriesDimensionRequest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r684813263
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
The index of `SeriesData['dimensions']` and `SeriesData['getDimension']` is not the dimensionIndex that users known, but the internal dimension index if dimension count > 30.
But there are someplace output that index to users or use them directly on `SeriesData['dimensions']`:
+ in `CustomView`: `wrapEncodeDef(customSeries.getData())`),
or use user input dimension index on `SeriesData['dimensions']`/`SeriesData['getDimension']`:
+ in `Customview`: `value(dim?: DimensionLoose, ...)`
+ in `component/marker/markerHelper.ts` L154 `data.getDimension(item.valueIndex)`. The valueIndex is specified by users.
+ in `component/visualMap/VisualMapModel.ts` L 390 `return data.getDimension(optDim);`.
+ ... (some other places that `SeriesData['getDimension']` is used).
Some bugs that we already have before this PR:
+ `RadarView` has bug if use `encode` before this PR:
```js
encode: {
value: [1, 2, 3, 4, 5, 6],
itemName: 7
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673621153
##########
File path: src/model/Series.ts
##########
@@ -385,14 +391,22 @@ class SeriesModel<Opt extends SeriesOption = SeriesOption> extends ComponentMode
inner(this).data = data;
}
- getSource(): Source {
- return inner(this).sourceManager.getSource();
+ getEncode() {
+ const encode = (this as Model<SeriesEncodeOptionMixin>).get('encode');
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And always use `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDataSchema` for each series. It contains the info generated by `createDimension` (like (1) dimensionDefineList, (2) whether dimension omitted, (3) source for this series, (4) ...).
3.1. `sourceManager` use `seriesDataSchema` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDataSchema`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDataSchema`.
3.3. `seriesData` use `seriesDataSchema` to init its dimensions, and use `seriesDataSchema` query dimName by dimIndex from source, or query dimIndex by dimName from source (only when "dimension omit" happen). If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDataSchema` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDataSchema` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
## About `SeriesDataSchema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675299767
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
My mistake, seems dimensionDefine will be undefined if `dimensions` is not given.
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
My mistake, seems `dimensionDefine` will be undefined if `dimensions` is not given.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r674516096
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675295921
##########
File path: src/chart/helper/createSeriesDataFromArray.ts
##########
@@ -0,0 +1,208 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+import * as zrUtil from 'zrender/src/core/util';
+import SeriesData from '../../data/SeriesData';
+import createDimensions, { CreateDimensionsParams, getDimCount } from '../../data/helper/createDimensions';
+import {getDimensionTypeByAxis} from '../../data/helper/dimensionHelper';
+import {getDataItemValue} from '../../util/model';
+import CoordinateSystem from '../../core/CoordinateSystem';
+import {getCoordSysInfoBySeries} from '../../model/referHelper';
+import { createSourceFromSeriesDataOption, isSourceInstance, Source } from '../../data/Source';
+import {enableDataStack} from '../../data/helper/dataStackHelper';
+import {makeSeriesEncodeForAxisCoordSys} from '../../data/helper/sourceHelper';
+import {
+ SOURCE_FORMAT_ORIGINAL, DimensionDefinitionLoose, DimensionDefinition, OptionSourceData, EncodeDefaulter
+} from '../../util/types';
+import SeriesModel from '../../model/Series';
+import DataStorage from '../../data/DataStorage';
+import DataDimensionInfo from '../../data/DataDimensionInfo';
+
+function isDataStorage(val: unknown): val is DataStorage {
+ return val instanceof DataStorage;
+}
+
+function getCoordSysDimDefs(
+ seriesModel: SeriesModel,
+ coordSysInfo: ReturnType<typeof getCoordSysInfoBySeries>
+) {
+ const coordSysName = seriesModel.get('coordinateSystem');
+ const registeredCoordSys = CoordinateSystem.get(coordSysName);
+
+ let coordSysDimDefs: DimensionDefinitionLoose[];
+
+ if (coordSysInfo && coordSysInfo.coordSysDims) {
+ coordSysDimDefs = zrUtil.map(coordSysInfo.coordSysDims, function (dim) {
+ const dimInfo = {
+ name: dim
+ } as DimensionDefinition;
+ const axisModel = coordSysInfo.axisMap.get(dim);
+ if (axisModel) {
+ const axisType = axisModel.get('type');
+ dimInfo.type = getDimensionTypeByAxis(axisType);
+ }
+ return dimInfo;
+ });
+ }
+
+ if (!coordSysDimDefs) {
+ // Get dimensions from registered coordinate system
+ coordSysDimDefs = (registeredCoordSys && (
+ registeredCoordSys.getDimensionsInfo
+ ? registeredCoordSys.getDimensionsInfo()
+ : registeredCoordSys.dimensions.slice()
+ )) || ['x', 'y'];
+ }
+
+ return coordSysDimDefs;
+}
+
+function injectOrdinalMeta(
+ dimInfoList: DataDimensionInfo[],
+ createInvertedIndices: boolean,
+ coordSysInfo: ReturnType<typeof getCoordSysInfoBySeries>
+) {
+ let firstCategoryDimIndex: number;
+ let hasNameEncode: boolean;
+ coordSysInfo && zrUtil.each(dimInfoList, function (dimInfo, dimIndex) {
+ const coordDim = dimInfo.coordDim;
+ const categoryAxisModel = coordSysInfo.categoryAxisMap.get(coordDim);
+ if (categoryAxisModel) {
+ if (firstCategoryDimIndex == null) {
+ firstCategoryDimIndex = dimIndex;
+ }
+ dimInfo.ordinalMeta = categoryAxisModel.getOrdinalMeta();
+ if (createInvertedIndices) {
+ dimInfo.createInvertedIndices = true;
+ }
+ }
+ if (dimInfo.otherDims.itemName != null) {
+ hasNameEncode = true;
+ }
+ });
+ if (!hasNameEncode && firstCategoryDimIndex != null) {
+ dimInfoList[firstCategoryDimIndex].otherDims.itemName = 0;
+ }
+ return firstCategoryDimIndex;
+}
+
+function createListFromArray(
+ sourceOrStore: Source | OptionSourceData | DataStorage,
+ seriesModel: SeriesModel,
+ opt?: {
+ generateCoord?: string
+ useEncodeDefaulter?: boolean | EncodeDefaulter
+ // By default: auto. If `true`, create inverted indices for all ordinal dimension on coordSys.
+ createInvertedIndices?: boolean
+ }
+): SeriesData {
+ opt = opt || {};
+
+ if (!isSourceInstance(sourceOrStore) && !isDataStorage(sourceOrStore)) {
+ sourceOrStore = createSourceFromSeriesDataOption(
+ sourceOrStore as OptionSourceData
+ );
+ }
+
+ const source = isDataStorage(sourceOrStore) ? sourceOrStore.getSource() : sourceOrStore;
+ const coordSysInfo = getCoordSysInfoBySeries(seriesModel);
+ const coordSysDimDefs = getCoordSysDimDefs(seriesModel, coordSysInfo);
+ const useEncodeDefaulter = opt.useEncodeDefaulter;
+
+ // Try to ignore unsed dimensions if sharing a high dimension datastorage
+ // 30 is an experience value.
+ const omitUnusedDimensions = isDataStorage(sourceOrStore) && sourceOrStore.getDimensionCount() > 30;
+ const encodeDefaulter = zrUtil.isFunction(useEncodeDefaulter)
+ ? useEncodeDefaulter
+ : useEncodeDefaulter
+ ? zrUtil.curry(makeSeriesEncodeForAxisCoordSys, coordSysDimDefs, seriesModel)
+ : null;
+
+ const createDimensionOptions = {
+ coordDimensions: coordSysDimDefs,
+ generateCoord: opt.generateCoord,
+ encodeDefine: seriesModel.getEncode()
+ // NOTE: If we call createDimensions on same source multiple times.
+ // It will break the encodeDefaulter which has sideeffects.
+ // So we prepare the default encode here instead of passing encoderDefaulter function.
+ || (encodeDefaulter && encodeDefaulter(
+ source, getDimCount(source, coordSysDimDefs, source.dimensionsDefine || [])
+ )),
+ omitUnusedDimensions
+ };
+ let dimInfoList = createDimensions(sourceOrStore, createDimensionOptions);
+ let firstCategoryDimIndex = injectOrdinalMeta(dimInfoList, opt.createInvertedIndices, coordSysInfo);
+
+ if (omitUnusedDimensions) {
+ // sourceOrStore
+ if (!(sourceOrStore as DataStorage).syncDimensionTypes(dimInfoList)) {
+ // We need a full dimensions list if we want to recreate the storage.
+ // Or the storage will ignore the data of other dimensions.
+ dimInfoList = createDimensions(sourceOrStore, zrUtil.extend(createDimensionOptions, {
+ omitUnusedDimensions: true
Review comment:
Fixed.
In fact, after fixing this bug, I noticed stacked high dimension data still has a serious performance issue because the stack dimensions will cause `canUse`(syncDimensionsInfo previously) to fail and DataStorage creation. This makes me have to rethink the design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r674490745
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
> Do we need to distinguish user defined string 'null' or 'undefined' from null / undefined here?
I don't understand what this mean. We should support user-defined 'null' / 'undefined' to be parsed to null / undefined?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r674500747
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
In this case. Dataset will still generate `dimensionsDefine` for the source. `dimensionsDefine` is only undefined when the source is `series.data` instead of dataset. In which case sharing storage is meaningless.
Going to add comments to explain it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-885382921
@100pah Thanks for the very detailed review. I rethink some of the designs again from your review and did some changes.
Not prepare data storage ahead of `createDimensions`. Instead, we first prepare dimensions in `createSeriesData`(renamed from `createListFromArray`), then pass the dimension to `SourceManager#getSharedDataStorage`(renamed from `SourceManager#getDataStorage`). It will generate a hash from the `type` and `ordinalMeta` in the dimensions. And using this hash to get and cache the DataStorage.
In this way, `DataStorage` can have enough dimension information for creation. And `SeriesData` can avoid validation and recreation of `DataStorage`.
If we don't use `createSeriesData`(for example in pie it uses createSeriesDataSimply). The strategy is the same as the master branch. We use `getSource` to get the `Source` instance(now it won't return DataStorage, which is a bit weird) and `SeriesData` will be responsible to initialize the DataStorage from this Source.
So basically this change makes DataStorage sharing not as aggressive as previous. We deal some of the sharing logic in the helper function in `createSeriesData` and use the name `SourceManager#getSharedDataStorage` to make it more clear. It can be expanded to `createSeriesDataSimply` if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] echarts-bot[bot] commented on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
echarts-bot[bot] commented on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-880401034
Thanks for your contribution!
The community will review it ASAP. In the meanwhile, please checkout [the coding standard](https://echarts.apache.org/en/coding-standard.html) and Wiki about [How to make a pull request](https://github.com/apache/echarts/wiki/How-to-make-a-pull-request).
The pull request is marked to be `PR: author is committer` because you are a committer of this project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673737621
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,79 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
Review comment:
Oh, got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675299767
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
My mistake, seems `dimensionDefine` will be undefined if `dimensions` is not given. In this case, seems the dimensions of these two series have the same name. Can we always generate dimensions names for dataset?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And always use `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like (1) dimensionDefineList, (2) whether dimension omitted, (3) source for this series, (4) ...).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source (only when "dimension omit" happen). If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
## About `SeriesDimensionRequest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And always use `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDimensionRequest` for each series. It contains the info generated by `createDimension` (like (1) dimensionDefineList, (2) whether dimension omitted, (3) source for this series, (4) ...).
3.1. `sourceManager` use `seriesDimensionRequest` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDimensionRequest`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDimensionRequest`.
3.3. `seriesData` use `seriesDimensionRequest` to init its dimensions, and use `seriesDimensionRequest` query dimName by dimIndex from source, or query dimIndex by dimName from source (only when "dimension omit" happen). If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDimensionRequest` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDimensionRequest` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
## About `SeriesDimensionRequest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah edited a comment on pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah edited a comment on pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#issuecomment-902228124
Some changes has been made in dffeb3a9a
## There are these issues existing before this commit:
1. If no dimensions specified on dataset, series can not really share one storage instance. (Because the each series will create its own dimension name (like ['x', 'y', 'value'], ['x', 'value', 'y'], and the storage hash is based on those names. So the hash can not match).
2. Each time `setOption` update series (but not change `dataset`), new data stack dimensions (and corresponding chunks) will be keep added to the shared data storage, and then the chunks will be more and more.
3. When "unused dimension omit" happen, the index of SeriesData['dimensions'] and SeriesData['getDimension'] is not the dimensionIndex that users known. But there are someplace still use dimensionIndex to visit them. (especially in visualMap and other similar cases that user can input dimension index via option).
4. If user only specify only type but no name in dimensions, their will be some bug when "unused dimension omit" happen. (Because unused dimensions will not auto-generate dimension name by `createDimensions` and so that it has no dimension name in storage, and can not be queried by dimension name).
## This commit try resolve those issues by this way:
1. Do not save the "dimName->dimIndex map" in data storage any more. Because:
1. In fact data storage do not need this map to read/write data.
2. dimNames are usually created based on each series info (like ['x', 'y', 'value'], ['x', 'value', 'y'], ...) if not specified by user. So they are different between series. But even those series have different generated dimension names, they can still share one storage, because essentially they visit the same dataset source by dimIndex.
2. Make `SeriesDimensionDefine` (that is, each item of `SeriesData['dimensionInfos']`) contain `storageDimensionIndex` to indicate its corresponding data store dimension index. And always use `storageDimensionIndex` to visit dat storage rather than dimName. `storageDimensionIndex` is created in `createDimension`.
3. create a new structure `SeriesDataSchema` for each series. It contains the info generated by `createDimension` (like (1) dimensionDefineList, (2) whether dimension omitted, (3) source for this series, (4) ...).
3.1. `sourceManager` use `seriesDataSchema` to find the shared storage by generate storage dimensions and hash based on the `dimCount` and `dimensionDefineList` (which are created by `createDimension` and saved in `seriesDataSchema`).
3.2. `dataStack` add "data stack dimensions" to `dimensionDefineList` in `seriesDataSchema`.
3.3. `seriesData` use `seriesDataSchema` to init its dimensions, and use `seriesDataSchema` query dimName by dimIndex from source, or query dimIndex by dimName from source (only when "dimension omit" happen). If different series option specify its own `dimensions` but share one `dataset`, the `source` get by `sourceManager` is different `source` instances in each series. Those `source` instances contain different `dimensionDefine` but reference the same "raw data". The data storage generated based on the "raw data" can be shared between series, but the `dimensionDefine` should not be shared. `SeriesDataSchema` encapsulate these mess and work each series to query dimension name or index.
3.4. `seriesDataSchema` do not create new data structure as possible as it can, but reference shared data structure (like `source` instance). So it will not cost memory issue.
4. Change the previous `storage.appendDimension` to `storage.ensureCalculationDimension` for data stack. That is, if its dimension has been created, reuse it.
5. Remove previous `canUse` method. Whether a storage can be shared by series is all determined by hash. The hash is generated in two ways:
5.1. For source format "arrayRows" (i.e., [[12, 33], [55, 99], ...]), dimension name do not need to be added to hash, because this kind of data actually visited by index. If two series have different dimension name (like 'x', 'y') for single index, they also can share the storage.
5.2. For source format "objectRows" (i.e, [{a: 12, b: 33}, {b: 55, a: 99}, ...]), property name 'a', 'b' will be added to hash, because this kind of data actually visited by property name.
5.3. And as before, dimension type, ordinal meta id, source header, series layout will also be added to hash.
6. Make `DataStorage` method immutable:
`DataStorage['filterSelf']` -> `DataStorage['filter']`
`DataStorage['selectRange']`
## PENDING
1. Should deprecate `dimName = seriesData.getDimension(dimLoose)` and `series.get(dimName)` but always use `dimIdx = seriesData.getDimensionIndex(dimLoose)` and `dataStorage.get(dimIdx)` instead. For examples:
```js
// Previously
const val = data.get(data.getDimension(dim), dataIdx);
// Now
const val = data.getStorage().get(data.getDimensionIndex(dim), dataIdx);
```
`seriesData.getDimension(dimLoose)` has a feature that convert dimIdx to dimName, which is not essentially necessary (because dimIdx can be used to visit data directly), but this feature require a "dimIdx->dimName map" in `SeriesData` (why? because when some dimensions are omitted, we can not use dimIdx on `SeriesData['dimensions']` directly).
2. Radar has bug when using `series.encode`. This commit do not fix the bug but keep as it is.
## Test case
<test/dataset-case.html>
<test/dataset-performance.html>
## About `SeriesDataSchema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673776591
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
1. Do we need to distinguish user defined string `'null'` or `'undefined'` from `null` / `undefined` here?
2. When user give duplicated dimension names, like `[{name: 'aa'}, {name: 'aa'}]`, previous logic resolve the duplication and the final result would be `[{name: 'aa'}, {name: 'aa1'}]`, but this line seems to cause the final result still duplicated as `[{name: 'aa'}, {name: 'aa'}]`. I am not sure whether duplicated dimension name will cause some issue.
##########
File path: src/data/SeriesData.ts
##########
@@ -0,0 +1,1338 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+/**
+ * List for data storage
+ */
+
+import * as zrUtil from 'zrender/src/core/util';
+import {PathStyleProps} from 'zrender/src/graphic/Path';
+import Model from '../model/Model';
+import DataDiffer from './DataDiffer';
+import {DataProvider, DefaultDataProvider} from './helper/dataProvider';
+import {summarizeDimensions, DimensionSummary} from './helper/dimensionHelper';
+import DataDimensionInfo from './DataDimensionInfo';
+import {ArrayLike, Dictionary, FunctionPropertyNames} from 'zrender/src/core/types';
+import Element from 'zrender/src/Element';
+import {
+ DimensionIndex, DimensionName, DimensionLoose, OptionDataItem,
+ ParsedValue, ParsedValueNumeric, DimensionUserOuput,
+ ModelOption, SeriesDataType, OptionSourceData, SOURCE_FORMAT_TYPED_ARRAY, SOURCE_FORMAT_ORIGINAL,
+ DecalObject,
+ OrdinalNumber
+} from '../util/types';
+import {convertOptionIdName, isDataItemOption} from '../util/model';
+import { setCommonECData } from '../util/innerStore';
+import type Graph from './Graph';
+import type Tree from './Tree';
+import type { VisualMeta } from '../component/visualMap/VisualMapModel';
+import {isSourceInstance, Source} from './Source';
+import { LineStyleProps } from '../model/mixin/lineStyle';
+import DataStorage, { DimValueGetter } from './DataStorage';
+
+const isObject = zrUtil.isObject;
+const map = zrUtil.map;
+
+const CtorInt32Array = typeof Int32Array === 'undefined' ? Array : Int32Array;
+
+// Use prefix to avoid index to be the same as otherIdList[idx],
+// which will cause weird udpate animation.
+const ID_PREFIX = 'e\0\0';
+
+const INDEX_NOT_FOUND = -1;
+
+type NameRepeatCount = {[name: string]: number};
+type ItrParamDims = DimensionLoose | Array<DimensionLoose>;
+// If Ctx not specified, use List as Ctx
+type CtxOrList<Ctx> = unknown extends Ctx ? SeriesData : Ctx;
+type EachCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => void;
+type EachCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => void;
+type EachCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => void;
+type FilterCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => boolean;
+type FilterCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => boolean;
+type FilterCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => boolean;
+type FilterCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => boolean;
+type MapArrayCb0<Ctx> = (this: CtxOrList<Ctx>, idx: number) => any;
+type MapArrayCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => any;
+type MapArrayCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) => any;
+type MapArrayCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => any;
+type MapCb1<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, idx: number) => ParsedValue | ParsedValue[];
+type MapCb2<Ctx> = (this: CtxOrList<Ctx>, x: ParsedValue, y: ParsedValue, idx: number) =>
+ ParsedValue | ParsedValue[];
+type MapCb<Ctx> = (this: CtxOrList<Ctx>, ...args: any) => ParsedValue | ParsedValue[];
+
+const TRANSFERABLE_PROPERTIES = [
+ 'hasItemOption', '_nameList', '_idList', '_invertedIndicesMap',
+ '_rawData', '_dimValueGetter',
+ '_nameDimIdx', '_idDimIdx', '_nameRepeatCount'
+];
+const CLONE_PROPERTIES = [
+ '_approximateExtent'
+];
+
+export interface DefaultDataVisual {
+ style: PathStyleProps
+ // Draw type determined which prop should be set with encoded color.
+ // It's only available on the global visual. Use getVisual('drawType') to access it.
+ // It will be set in visual/style.ts module in the first priority.
+ drawType: 'fill' | 'stroke'
+
+ symbol?: string
+ symbolSize?: number | number[]
+ symbolRotate?: number
+ symbolKeepAspect?: boolean
+ symbolOffset?: string | number | (string | number)[]
+
+ liftZ?: number
+ // For legend.
+ legendIcon?: string
+ legendLineStyle?: LineStyleProps
+
+ // visualMap will inject visualMeta data
+ visualMeta?: VisualMeta[]
+
+ // If color is encoded from palette
+ colorFromPalette?: boolean
+
+ decal?: DecalObject
+}
+
+export interface DataCalculationInfo<SERIES_MODEL> {
+ stackedDimension: string;
+ stackedByDimension: string;
+ isStackedByIndex: boolean;
+ stackedOverDimension: string;
+ stackResultDimension: string;
+ stackedOnSeries?: SERIES_MODEL;
+}
+
+// -----------------------------
+// Internal method declarations:
+// -----------------------------
+let prepareInvertedIndex: (data: SeriesData) => void;
+let getId: (data: SeriesData, rawIndex: number) => string;
+let getIdNameFromStore: (data: SeriesData, dimIdx: number, dataIdx: number) => string;
+let normalizeDimensions: (dimensions: ItrParamDims) => Array<DimensionLoose>;
+let validateDimensions: (data: SeriesData, dims: DimensionIndex[]) => void;
+let transferProperties: (target: SeriesData, source: SeriesData) => void;
+let cloneListForMapAndSample: (original: SeriesData) => SeriesData;
+let makeIdFromName: (data: SeriesData, idx: number) => void;
+
+class SeriesData<
+ HostModel extends Model = Model,
+ Visual extends DefaultDataVisual = DefaultDataVisual
+> {
+
+ readonly type = 'list';
+
+ readonly dimensions: string[];
+
+ // Infomation of each data dimension, like data type.
+ private _dimensionInfos: {[dimName: string]: DataDimensionInfo};
+
+ readonly hostModel: HostModel;
+
+ /**
+ * @readonly
+ */
+ dataType: SeriesDataType;
+
+ /**
+ * @readonly
+ * Host graph if List is used to store graph nodes / edges.
+ */
+ graph?: Graph;
+
+ /**
+ * @readonly
+ * Host tree if List is used to store tree ndoes.
+ */
+ tree?: Tree;
+
+ private _store: DataStorage;
+
+ private _nameList: string[] = [];
+ private _idList: string[] = [];
+
+ // Models of data option is stored sparse for optimizing memory cost
+ // Never used yet (not used yet).
+ // private _optionModels: Model[] = [];
+
+ // Global visual properties after visual coding
+ private _visual: Dictionary<any> = {};
+
+ // Globel layout properties.
+ private _layout: Dictionary<any> = {};
+
+ // Item visual properties after visual coding
+ private _itemVisuals: Dictionary<any>[] = [];
+
+ // Item layout properties after layout
+ private _itemLayouts: any[] = [];
+
+ // Graphic elemnents
+ private _graphicEls: Element[] = [];
+
+ // key: dim, value: extent
+ private _approximateExtent: {[dimName: string]: [number, number]} = {};
+
+ private _dimensionsSummary: DimensionSummary;
+
+ private _invertedIndicesMap: {[dimName: string]: ArrayLike<number>};
+
+ private _calculationInfo: DataCalculationInfo<HostModel> = {} as DataCalculationInfo<HostModel>;
+
+ // User output info of this data.
+ // DO NOT use it in other places!
+ // When preparing user params for user callbacks, we have
+ // to clone these inner data structures to prevent users
+ // from modifying them to effect built-in logic. And for
+ // performance consideration we make this `userOutput` to
+ // avoid clone them too many times.
+ readonly userOutput: DimensionUserOuput;
+
+ // Having detected that there is data item is non primitive type
+ // (in type `OptionDataItemObject`).
+ // Like `data: [ { value: xx, itemStyle: {...} }, ...]`
+ // At present it only happen in `SOURCE_FORMAT_ORIGINAL`.
+ hasItemOption: boolean = false;
+
+ // id or name is used on dynamic data, mapping old and new items.
+ // When generating id from name, avoid repeat.
+ private _nameRepeatCount: NameRepeatCount;
+ private _nameDimIdx: number;
+ private _idDimIdx: number;
+
+ private __wrappedMethods: string[];
+
+ // Methods that create a new list based on this list should be listed here.
+ // Notice that those method should `RETURN` the new list.
+ TRANSFERABLE_METHODS = ['cloneShallow', 'downSample', 'lttbDownSample', 'map'] as const;
+ // Methods that change indices of this list should be listed here.
+ CHANGABLE_METHODS = ['filterSelf', 'selectRange'] as const;
+ DOWNSAMPLE_METHODS = ['downSample', 'lttbDownSample'] as const;
+
+ /**
+ * @param dimensions
+ * For example, ['someDimName', {name: 'someDimName', type: 'someDimType'}, ...].
+ * Dimensions should be concrete names like x, y, z, lng, lat, angle, radius
+ */
+ constructor(dimensions: Array<string | object | DataDimensionInfo>, hostModel: HostModel) {
+ dimensions = dimensions || ['x', 'y'];
+
+ const dimensionInfos: Dictionary<DataDimensionInfo> = {};
+ const dimensionNames = [];
+ const invertedIndicesMap: Dictionary<number[]> = {};
+
+ for (let i = 0; i < dimensions.length; i++) {
+ // Use the original dimensions[i], where other flag props may exists.
+ const dimInfoInput = dimensions[i];
+
+ const dimensionInfo: DataDimensionInfo =
+ zrUtil.isString(dimInfoInput)
+ ? new DataDimensionInfo({name: dimInfoInput})
+ : !(dimInfoInput instanceof DataDimensionInfo)
+ ? new DataDimensionInfo(dimInfoInput)
+ : dimInfoInput;
+
+ const dimensionName = dimensionInfo.name;
+ dimensionInfo.type = dimensionInfo.type || 'float';
+ if (!dimensionInfo.coordDim) {
+ dimensionInfo.coordDim = dimensionName;
+ dimensionInfo.coordDimIndex = 0;
+ }
+
+ const otherDims = dimensionInfo.otherDims = dimensionInfo.otherDims || {};
+ dimensionNames.push(dimensionName);
+ dimensionInfos[dimensionName] = dimensionInfo;
+
+ dimensionInfo.index = i;
+
+ if (dimensionInfo.createInvertedIndices) {
+ invertedIndicesMap[dimensionName] = [];
+ }
+ if (otherDims.itemName === 0) {
+ this._nameDimIdx = i;
+ }
+ if (otherDims.itemId === 0) {
+ this._idDimIdx = i;
+ }
+ }
+
+ this.dimensions = dimensionNames;
+ this._dimensionInfos = dimensionInfos;
+ this.hostModel = hostModel;
+
+ // Cache summary info for fast visit. See "dimensionHelper".
+ this._dimensionsSummary = summarizeDimensions(this);
+
+ this._invertedIndicesMap = invertedIndicesMap;
+
+ this.userOutput = this._dimensionsSummary.userOutput;
+ }
+
+ /**
+ * The meanings of the input parameter `dim`:
+ *
+ * + If dim is a number (e.g., `1`), it means the index of the dimension.
+ * For example, `getDimension(0)` will return 'x' or 'lng' or 'radius'.
+ * + If dim is a number-like string (e.g., `"1"`):
+ * + If there is the same concrete dim name defined in `this.dimensions`, it means that concrete name.
+ * + If not, it will be converted to a number, which means the index of the dimension.
+ * (why? because of the backward compatbility. We have been tolerating number-like string in
+ * dimension setting, although now it seems that it is not a good idea.)
+ * For example, `visualMap[i].dimension: "1"` is the same meaning as `visualMap[i].dimension: 1`,
+ * if no dimension name is defined as `"1"`.
+ * + If dim is a not-number-like string, it means the concrete dim name.
+ * For example, it can be be default name `"x"`, `"y"`, `"z"`, `"lng"`, `"lat"`, `"angle"`, `"radius"`,
+ * or customized in `dimensions` property of option like `"age"`.
+ *
+ * Get dimension name
+ * @param dim See above.
+ * @return Concrete dim name.
+ */
+ getDimension(dim: DimensionLoose): DimensionName {
+ if (typeof dim === 'number'
+ // If being a number-like string but not being defined a dimension name.
+ || (!isNaN(dim as any) && !this._dimensionInfos.hasOwnProperty(dim))
+ ) {
+ dim = this.dimensions[dim as DimensionIndex];
+ }
+ return dim as DimensionName;
+ }
+
+ private _getStoreDimIndex(dim: DimensionLoose): DimensionIndex {
+ return this._store.getDimensionIndex(this.getDimension(dim));
+ }
+
+ /**
+ * Get type and calculation info of particular dimension
+ * @param dim
+ * Dimension can be concrete names like x, y, z, lng, lat, angle, radius
+ * Or a ordinal number. For example getDimensionInfo(0) will return 'x' or 'lng' or 'radius'
+ */
+ getDimensionInfo(dim: DimensionLoose): DataDimensionInfo {
+ // Do not clone, because there may be categories in dimInfo.
+ return this._dimensionInfos[this.getDimension(dim)];
+ }
+
+ /**
+ * concrete dimension name list on coord.
+ */
+ getDimensionsOnCoord(): DimensionName[] {
+ return this._dimensionsSummary.dataDimsOnCoord.slice();
+ }
+
+ /**
+ * @param coordDim
+ * @param idx A coordDim may map to more than one data dim.
+ * If not specified, return the first dim not extra.
+ * @return concrete data dim. If not found, return null/undefined
+ */
+ mapDimension(coordDim: DimensionName): DimensionName;
+ mapDimension(coordDim: DimensionName, idx: number): DimensionName;
+ mapDimension(coordDim: DimensionName, idx?: number): DimensionName {
+ const dimensionsSummary = this._dimensionsSummary;
+
+ if (idx == null) {
+ return dimensionsSummary.encodeFirstDimNotExtra[coordDim] as any;
+ }
+
+ const dims = dimensionsSummary.encode[coordDim];
+ return dims ? dims[idx as number] as any : null;
+ }
+
+ mapDimensionsAll(coordDim: DimensionName): DimensionName[] {
+ const dimensionsSummary = this._dimensionsSummary;
+ const dims = dimensionsSummary.encode[coordDim];
+ return (dims || []).slice();
+ }
+
+ getStorage() {
+ return this._store;
+ }
+
+ /**
+ * Initialize from data
+ * @param data source or data or data storage.
+ * @param nameList The name of a datum is used on data diff and
+ * default label/tooltip.
+ * A name can be specified in encode.itemName,
+ * or dataItem.name (only for series option data),
+ * or provided in nameList from outside.
+ */
+ initData(
+ data: Source | OptionSourceData | DataStorage | DataProvider,
+ nameList?: string[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ let store: DataStorage;
+ const dimensions = this.dimensions;
+ const dimensionInfos = map(dimensions, dimName => this._dimensionInfos[dimName]);
+ if (data instanceof DataStorage) {
+ if (data.syncDimensionTypes(dimensionInfos)) {
+ store = data;
+ }
+ // Sync failed
+ else {
+ // Needs to recreate data storage if it's not match the given dimension.
+ data = data.getSource();
+ }
+ }
+
+ if (!store) {
+ const provider = (isSourceInstance(data) || zrUtil.isArrayLike(data))
+ ? new DefaultDataProvider(data as Source | OptionSourceData, dimensions.length)
+ : data as DataProvider;
+ store = new DataStorage();
+ store.initData(provider, dimensionInfos, dimValueGetter);
+ }
+
+ this._store = store;
+
+ // Reset
+ this._nameList = (nameList || []).slice();
+ this._idList = [];
+ this._nameRepeatCount = {};
+
+ this._doInit(0, store.count());
+ }
+
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ */
+ appendData(data: ArrayLike<any>): void {
+ const range = this._store.appendData(data);
+ this._doInit(range[0], range[1]);
+ }
+ /**
+ * Caution: Can be only called on raw data (before `this._indices` created).
+ * This method does not modify `rawData` (`dataProvider`), but only
+ * add values to storage.
+ *
+ * The final count will be increased by `Math.max(values.length, names.length)`.
+ *
+ * @param values That is the SourceType: 'arrayRows', like
+ * [
+ * [12, 33, 44],
+ * [NaN, 43, 1],
+ * ['-', 'asdf', 0]
+ * ]
+ * Each item is exaclty cooresponding to a dimension.
+ */
+ appendValues(values: any[][], names?: string[]): void {
+ const {start, end} = this._store.appendValues(values, names.length);
+ const shouldMakeIdFromName = this._shouldMakeIdFromName();
+
+ this._updateOrdinalMeta();
+
+ if (names) {
+ for (let idx = start; idx < end; idx++) {
+ const sourceIdx = idx - start;
+ this._nameList[idx] = names[sourceIdx];
+ if (shouldMakeIdFromName) {
+ makeIdFromName(this, idx);
+ }
+ }
+ }
+ }
+
+ private _updateOrdinalMeta() {
+ const store = this._store;
+ const dimensions = this.dimensions;
+ for (let i = 0; i < dimensions.length; i++) {
+ const dimInfo = this._dimensionInfos[dimensions[i]];
+ if (dimInfo.ordinalMeta) {
+ const dimIdx = store.getDimensionIndex(dimensions[i]);
+ store.collectOrdinalMeta(dimIdx, dimInfo.ordinalMeta);
+ }
+ }
+ }
+
+ private _shouldMakeIdFromName() {
+ const provider = this._store.getProvider();
+ return this._idDimIdx == null
+ && provider.getSource().sourceFormat !== SOURCE_FORMAT_TYPED_ARRAY
+ && !provider.fillStorage;
+ }
+
+ private _doInit(start: number, end: number): void {
+ if (start >= end) {
+ return;
+ }
+
+ const store = this._store;
+ const provider = store.getProvider();
+
+ this._updateOrdinalMeta();
+
+ const nameList = this._nameList;
+ const idList = this._idList;
+ const sourceFormat = provider.getSource().sourceFormat;
+ const isFormatOriginal = sourceFormat === SOURCE_FORMAT_ORIGINAL;
+
+ // Each data item is value
+ // [1, 2]
+ // 2
+ // Bar chart, line chart which uses category axis
+ // only gives the 'y' value. 'x' value is the indices of category
+ // Use a tempValue to normalize the value to be a (x, y) value
+ // If dataItem is {name: ...} or {id: ...}, it has highest priority.
+ // This kind of ids and names are always stored `_nameList` and `_idList`.
+ if (isFormatOriginal && !provider.pure) {
+ const sharedDataItem = [] as OptionDataItem;
+ for (let idx = start; idx < end; idx++) {
+ // NOTICE: Try not to write things into dataItem
+ const dataItem = provider.getItem(idx, sharedDataItem);
+ if (!this.hasItemOption && isDataItemOption(dataItem)) {
+ this.hasItemOption = true;
+ }
+ if (dataItem) {
+ const itemName = (dataItem as any).name;
+ if (nameList[idx] == null && itemName != null) {
+ nameList[idx] = convertOptionIdName(itemName, null);
+ }
+ const itemId = (dataItem as any).id;
+ if (idList[idx] == null && itemId != null) {
+ idList[idx] = convertOptionIdName(itemId, null);
+ }
+ }
+ }
+ }
+
+ if (this._shouldMakeIdFromName()) {
+ for (let idx = start; idx < end; idx++) {
+ makeIdFromName(this, idx);
+ }
+ }
+
+ prepareInvertedIndex(this);
+ }
+ /**
+ * PENDING: In fact currently this function is only used to short-circuit
+ * the calling of `scale.unionExtentFromData` when data have been filtered by modules
+ * like "dataZoom". `scale.unionExtentFromData` is used to calculate data extent for series on
+ * an axis, but if a "axis related data filter module" is used, the extent of the axis have
+ * been fixed and no need to calling `scale.unionExtentFromData` actually.
+ * But if we add "custom data filter" in future, which is not "axis related", this method may
+ * be still needed.
+ *
+ * Optimize for the scenario that data is filtered by a given extent.
+ * Consider that if data amount is more than hundreds of thousand,
+ * extent calculation will cost more than 10ms and the cache will
+ * be erased because of the filtering.
+ */
+ getApproximateExtent(dim: DimensionLoose): [number, number] {
+ return this._approximateExtent[dim] || this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ /**
+ * Calculate extent on a filtered data might be time consuming.
+ * Approximate extent is only used for: calculte extent of filtered data outside.
+ */
+ setApproximateExtent(extent: [number, number], dim: DimensionLoose): void {
+ dim = this.getDimension(dim);
+ this._approximateExtent[dim] = extent.slice() as [number, number];
+ }
+
+ getCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY
+ ): DataCalculationInfo<HostModel>[CALC_INFO_KEY] {
+ return this._calculationInfo[key];
+ }
+
+ /**
+ * @param key or k-v object
+ */
+ setCalculationInfo(
+ key: DataCalculationInfo<HostModel>
+ ): void;
+ setCalculationInfo<CALC_INFO_KEY extends keyof DataCalculationInfo<HostModel>>(
+ key: CALC_INFO_KEY,
+ value: DataCalculationInfo<HostModel>[CALC_INFO_KEY]
+ ): void;
+ setCalculationInfo(
+ key: (keyof DataCalculationInfo<HostModel>) | DataCalculationInfo<HostModel>,
+ value?: DataCalculationInfo<HostModel>[keyof DataCalculationInfo<HostModel>]
+ ): void {
+ isObject(key)
+ ? zrUtil.extend(this._calculationInfo, key as object)
+ : ((this._calculationInfo as any)[key] = value);
+ }
+
+ /**
+ * @return Never be null/undefined. `number` will be converted to string. Becuase:
+ * In most cases, name is used in display, where returning a string is more convenient.
+ * In other cases, name is used in query (see `indexOfName`), where we can keep the
+ * rule that name `2` equals to name `'2'`.
+ */
+ getName(idx: number): string {
+ const rawIndex = this.getRawIndex(idx);
+ let name = this._nameList[rawIndex];
+ if (name == null && this._nameDimIdx != null) {
+ name = getIdNameFromStore(this, this._nameDimIdx, rawIndex);
+ }
+ if (name == null) {
+ name = '';
+ }
+ return name;
+ }
+
+ private _getCategory(dimIdx: number, idx: number) {
+ const ordinal = this._store.get(dimIdx, idx);
+ const ordinalMeta = this._store.getOrdinalMeta(dimIdx);
+ if (ordinalMeta) {
+ return ordinalMeta.categories[ordinal as OrdinalNumber];
+ }
+ return ordinal;
+ }
+
+ /**
+ * @return Never null/undefined. `number` will be converted to string. Becuase:
+ * In all cases having encountered at present, id is used in making diff comparison, which
+ * are usually based on hash map. We can keep the rule that the internal id are always string
+ * (treat `2` is the same as `'2'`) to make the related logic simple.
+ */
+ getId(idx: number): string {
+ return getId(this, this.getRawIndex(idx));
+ }
+
+ count() {
+ return this._store.count();
+ }
+ /**
+ * Get value. Return NaN if idx is out of range.
+ * @param dim Dim must be concrete name.
+ */
+ get(dim: DimensionName, idx: number): ParsedValue {
+ const store = this._store;
+ return store.get(store.getDimensionIndex(dim), idx);
+ }
+
+ getByRawIndex(dim: DimensionName, rawIdx: number): ParsedValue {
+ const store = this._store;
+ return store.getByRawIndex(store.getDimensionIndex(dim), rawIdx);
+ }
+
+ getIndices() {
+ return this._store.getIndices();
+ }
+
+ getDataExtent(dim: DimensionLoose): [number, number] {
+ return this._store.getDataExtent(this._getStoreDimIndex(dim));
+ }
+
+ getSum(dim: DimensionLoose): number {
+ return this._store.getSum(this._getStoreDimIndex(dim));
+ }
+
+ getMedian(dim: DimensionLoose): number {
+ return this._store.getMedian(this._getStoreDimIndex(dim));
+ }
+ /**
+ * Get value for multi dimensions.
+ * @param dimensions If ignored, using all dimensions.
+ */
+ getValues(idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[], idx: number): ParsedValue[];
+ getValues(dimensions: readonly DimensionName[] | number, idx?: number): ParsedValue[] {
+ const store = this._store;
+ return zrUtil.isArray(dimensions)
+ ? store.getValues(map(dimensions, dim => this._getStoreDimIndex(dim)), idx)
+ : store.getValues(dimensions as number);
+ }
+
+ /**
+ * If value is NaN. Inlcuding '-'
+ * Only check the coord dimensions.
+ */
+ hasValue(idx: number): boolean {
+ const dataDimsOnCoord = this._dimensionsSummary.dataDimsOnCoord;
+ for (let i = 0, len = dataDimsOnCoord.length; i < len; i++) {
+ // Ordinal type originally can be string or number.
+ // But when an ordinal type is used on coord, it can
+ // not be string but only number. So we can also use isNaN.
+ if (isNaN(this.get(dataDimsOnCoord[i], idx) as any)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Retreive the index with given name
+ */
+ indexOfName(name: string): number {
+ for (let i = 0, len = this._store.count(); i < len; i++) {
+ if (this.getName(i) === name) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ getRawIndex(idx: number) {
+ return this._store.getRawIndex(idx);
+ }
+
+ indexOfRawIndex(rawIndex: number) {
+ return this._store.indexOfRawIndex(rawIndex);
+ }
+
+ /**
+ * Only support the dimension which inverted index created.
+ * Do not support other cases until required.
+ * @param dim concrete dim
+ * @param value ordinal index
+ * @return rawIndex
+ */
+ rawIndexOf(dim: DimensionName, value: OrdinalNumber): number {
+ const invertedIndices = dim && this._invertedIndicesMap[dim];
+ if (__DEV__) {
+ if (!invertedIndices) {
+ throw new Error('Do not supported yet');
+ }
+ }
+ const rawIndex = invertedIndices[value];
+ if (rawIndex == null || isNaN(rawIndex)) {
+ return INDEX_NOT_FOUND;
+ }
+ return rawIndex;
+ }
+
+ /**
+ * Retreive the index of nearest value
+ * @param dim
+ * @param value
+ * @param [maxDistance=Infinity]
+ * @return If and only if multiple indices has
+ * the same value, they are put to the result.
+ */
+ indicesOfNearest(dim: DimensionLoose, value: number, maxDistance?: number): number[] {
+ return this._store.indicesOfNearest(
+ this._getStoreDimIndex(dim),
+ value, maxDistance
+ );
+ }
+ /**
+ * Data iteration
+ * @param ctx default this
+ * @example
+ * list.each('x', function (x, idx) {});
+ * list.each(['x', 'y'], function (x, y, idx) {});
+ * list.each(function (idx) {})
+ */
+ each<Ctx>(cb: EachCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): void;
+ each<Ctx>(dims: DimensionLoose, cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose], cb: EachCb1<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: EachCb2<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(dims: ItrParamDims, cb: EachCb<Ctx>, ctx?: Ctx): void;
+ each<Ctx>(
+ dims: ItrParamDims | EachCb<Ctx>,
+ cb: EachCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): void {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store.each(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ }
+ /**
+ * Data filter
+ */
+ filterSelf<Ctx>(cb: FilterCb0<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): this;
+ filterSelf<Ctx>(dims: DimensionLoose, cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose], cb: FilterCb1<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: FilterCb2<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(dims: ItrParamDims, cb: FilterCb<Ctx>, ctx?: Ctx): this;
+ filterSelf<Ctx>(
+ dims: ItrParamDims | FilterCb<Ctx>,
+ cb: FilterCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(normalizeDimensions(dims), this._getStoreDimIndex, this);
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ // Clone first
+ this._store = this._store.clone();
+ this._store.filterSelf(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+
+ return this;
+ }
+
+ /**
+ * Select data in range. (For optimization of filter)
+ * (Manually inline code, support 5 million data filtering in data zoom.)
+ */
+ selectRange(range: Record<string, [number, number]>): SeriesData {
+ 'use strict';
+
+ const innerRange: Record<number, [number, number]> = {};
+ const dims = zrUtil.keys(range);
+ const dimIndices: number[] = [];
+ zrUtil.each(dims, (dim) => {
+ const dimIdx = this._getStoreDimIndex(dim);
+ innerRange[dimIdx] = range[dim];
+ dimIndices.push(dimIdx);
+ });
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ this._store = this._store.clone();
+ this._store.selectRange(innerRange);
+ return this;
+ }
+
+ /**
+ * Data mapping to a plain array
+ */
+ mapArray<Ctx, Cb extends MapArrayCb0<Ctx>>(cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-disable */
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: DimensionLoose, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb1<Ctx>>(dims: [DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb2<Ctx>>(dims: [DimensionLoose, DimensionLoose], cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ mapArray<Ctx, Cb extends MapArrayCb<Ctx>>(dims: ItrParamDims, cb: Cb, ctx?: Ctx, ctxCompat?: Ctx): ReturnType<Cb>[];
+ /* eslint-enable */
+ mapArray<Ctx>(
+ dims: ItrParamDims | MapArrayCb<Ctx>,
+ cb: MapArrayCb<Ctx> | Ctx,
+ ctx?: Ctx
+ ): unknown[] {
+ 'use strict';
+
+ if (typeof dims === 'function') {
+ ctx = cb as Ctx;
+ cb = dims;
+ dims = [];
+ }
+
+ // ctxCompat just for compat echarts3
+ ctx = (ctx || this) as Ctx;
+
+ const result: unknown[] = [];
+ this.each(dims, function () {
+ result.push(cb && (cb as MapArrayCb<Ctx>).apply(this, arguments));
+ }, ctx);
+ return result;
+ }
+
+ /**
+ * Data mapping to a new List with given dimensions
+ */
+ map<Ctx>(dims: DimensionLoose, cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(dims: [DimensionLoose], cb: MapCb1<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ /* eslint-disable-next-line */
+ map<Ctx>(dims: [DimensionLoose, DimensionLoose], cb: MapCb2<Ctx>, ctx?: Ctx, ctxCompat?: Ctx): SeriesData<HostModel>;
+ map<Ctx>(
+ dims: ItrParamDims,
+ cb: MapCb<Ctx>,
+ ctx?: Ctx,
+ ctxCompat?: Ctx
+ ): SeriesData {
+ 'use strict';
+
+ // ctxCompat just for compat echarts3
+ const fCtx = (ctx || ctxCompat || this) as CtxOrList<Ctx>;
+
+ const dimIndices = map(
+ normalizeDimensions(dims), this._getStoreDimIndex, this
+ );
+
+ if (__DEV__) {
+ validateDimensions(this, dimIndices);
+ }
+
+ const list = cloneListForMapAndSample(this);
+ list._store = this._store.map(dimIndices, (fCtx
+ ? zrUtil.bind(cb as any, fCtx as any)
+ : cb) as any
+ );
+ return list;
+ }
+
+ /**
+ * Large data down sampling on given dimension
+ * @param sampleIndex Sample index for name and id
+ */
+ downSample(
Review comment:
### Just a programming pattern thinking
Since we have shared data in `DataStorage` instead of `SeriesData/List`,
should we change the interface like `SeriesData#downSamle`/`SeriesData#lttbDownSample`/`SeriesData#map` from immutable to mutable (only provide immutable interface in `DataStorage`),
And substitute `SeriesData#reset` for `SeriesData#cloneShallow`.
And remove `Series#setData`.
Finally a `series` instance will not change the reference of `seriesData` instance,
and the file `src/data/helper/linkSeriesData.ts`, which is a little weird, can be removed.
And the concept might be more simple: only `DataStorage` can be shared and immutable, rather than both `DataStorage` and `SeriesData` need to be immutable.
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
Review comment:
Should also add a delimiter between `source.startIndex` and dimensions.
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
If there is a dateset have 31 dimensions like:
```js
dataset: {
source: [
['year', 'a1', 'a2', ..., 'a30'], // dimensions
['2021', 23, 11, ..., 51], // data
...
]
}
```
And the series has encode:
```js
encode: {
x: 0,
y: 31
}
```
The result dimension will be:
```js
[
{name: 'year'},
{name: 'a30'} // finally map to the second column ('a1') of dataset.source,
// which is not correct
]
```
There is a test case show the issue above:
```js
// If dimIndexMax less then 30, the result is correct
// If dimIndexMax biggger then 30, the result will not be correct:
// series encode will not work
var dimIndexMax = 32;
var source = [
['year'],
['2021'],
['2022'],
];
for (var i = 0; i < dimIndexMax; i++) {
source[0].push('a' + (i + 1));
source[1].push(Math.random());
source[2].push(Math.random());
}
source[1][1] = 1;
source[2][1] = 9;
source[1][dimIndexMax] = 111;
source[2][dimIndexMax] = 999;
option = {
xAxis: {
type: 'category',
scale: true
},
yAxis: {},
dataset: {
source: source
},
tooltip: {},
series: {
type: 'scatter',
encode: {
x: 0,
y: dimIndexMax
}
}
};
```
I think if we omit unused dimensions, we might have to make the dimensions array "index-insensitve".
A "index-insensitve" dimension array require the change of "data-transform" API:
```js
var dimensions = upstream.cloneAllDimensionInfo()
for (var idx = 0; idx < dimensions.length; idx++) {
// idx means nothing. Not be the dim index any more.
var dimIdx = dimensions[i].index; // (or dimensions[i].dimensionIndex)
}
```
And some place manually construct `DataDimensionInfo` (like `echarts/src/data/helper/dataStackHelper.ts`) also need to provide dimension index in each item.
And if make "index-insensitve" dimension , the sort here is not needed.
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
If `dimensionsDefine` is not provided, should we consider the case (in future):
```js
dataset: {
source: [
[12, 55, ...],
[11, 85, ...],
...
]
},
series: [{
encode: {x: 0, y: 1},
}, {
encode: {x: 0, y: 2},
},
...
]
```
where totally no dimension specified, and here the `dimensionsDefine` is `undefined`, but it seems reasonable to adopt the feature of "share data store". But it seems still not an easy job, since we do lots of things based on dimension name but in the case the dimension name is generated by coord sys when series initData. Perhaps dimension index should be the first class instead of dimension name.
But also see the comment of `createDimensions.ts` L282.
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
Review comment:
`dataDimNameAutoIndex` should better always be `0`, or if this `resultItem.coordDim` does not equal to the previous `resultItem.coordDim`, `dataDimNameAutoIndex` go back to `0`.
Otherwise, if
```js
series: {
data: ...,
encode: {
x: [0, 1, 2],
y: 3
}
}
```
The generated dimension name will be:
```js
['x', 'x1', 'x2', 'y3']
```
rather than
```js
['x', 'x1', 'x2', 'y']
```
The letter one might be better?
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
Review comment:
There is a prerequisite: if `omitUnusedDimensions` is `true`, the `name` and `coordDim` will never be null/undefined here and no need to generate.
But that prerequisite is not obvious. It depend on the L411 in `sourceManager.ts`, where if no dimension given by user, the feature of "share data store" will not be enabled.
Should we give some comment here and in `sourceManager.ts` L411 ,. to hint this fact, in case forget it in future and cause potential buggy modification? Because the prerequisite might be change in future, see the comment in `sourceManager.ts` L411 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673629571
##########
File path: src/chart/helper/createSeriesDataFromArray.ts
##########
@@ -107,26 +99,92 @@ function createListFromArray(source: Source | OptionSourceData, seriesModel: Ser
if (!hasNameEncode && firstCategoryDimIndex != null) {
dimInfoList[firstCategoryDimIndex].otherDims.itemName = 0;
}
+ return firstCategoryDimIndex;
+}
- const stackCalculationInfo = enableDataStack(seriesModel, dimInfoList);
+function createListFromArray(
+ sourceOrStore: Source | OptionSourceData | DataStorage,
+ seriesModel: SeriesModel,
+ opt?: {
+ generateCoord?: string
+ useEncodeDefaulter?: boolean | EncodeDefaulter
+ // By default: auto. If `true`, create inverted indices for all ordinal dimension on coordSys.
+ createInvertedIndices?: boolean
+ }
+): SeriesData {
+ opt = opt || {};
- const list = new List(dimInfoList, seriesModel);
+ if (!isSourceInstance(sourceOrStore) && !isDataStorage(sourceOrStore)) {
+ sourceOrStore = createSourceFromSeriesDataOption(
+ sourceOrStore as OptionSourceData
+ );
+ }
- list.setCalculationInfo(stackCalculationInfo);
+ const source = isDataStorage(sourceOrStore) ? sourceOrStore.getSource() : sourceOrStore;
+ const coordSysInfo = getCoordSysInfoBySeries(seriesModel);
+ const coordSysDimDefs = getCoordSysDimDefs(seriesModel, coordSysInfo);
+ const useEncodeDefaulter = opt.useEncodeDefaulter;
- const dimValueGetter = (firstCategoryDimIndex != null && isNeedCompleteOrdinalData(source))
- ? function (this: List, itemOpt: any, dimName: string, dataIndex: number, dimIndex: number) {
- // Use dataIndex as ordinal value in categoryAxis
- return dimIndex === firstCategoryDimIndex
- ? dataIndex
- : this.defaultDimValueGetter(itemOpt, dimName, dataIndex, dimIndex);
- }
+ // Try to ignore unsed dimensions if sharing a high dimension datastorage
+ // 30 is an experience value.
+ const omitUnusedDimensions = isDataStorage(sourceOrStore) && sourceOrStore.getDimensionCount() > 30;
+ const encodeDefaulter = zrUtil.isFunction(useEncodeDefaulter)
+ ? useEncodeDefaulter
+ : useEncodeDefaulter
+ ? zrUtil.curry(makeSeriesEncodeForAxisCoordSys, coordSysDimDefs, seriesModel)
: null;
- list.hasItemOption = false;
- list.initData(source, null, dimValueGetter);
+ const createDimensionOptions = {
+ coordDimensions: coordSysDimDefs,
+ generateCoord: opt.generateCoord,
+ encodeDefine: seriesModel.getEncode()
+ // NOTE: If we call createDimensions on same source multiple times.
+ // It will break the encodeDefaulter which has sideeffects.
+ // So we prepare the default encode here instead of passing encoderDefaulter function.
+ || (encodeDefaulter && encodeDefaulter(
+ source, getDimCount(source, coordSysDimDefs, source.dimensionsDefine || [])
+ )),
+ omitUnusedDimensions
+ };
+ let dimInfoList = createDimensions(sourceOrStore, createDimensionOptions);
+ let firstCategoryDimIndex = injectOrdinalMeta(dimInfoList, opt.createInvertedIndices, coordSysInfo);
+
+ if (omitUnusedDimensions) {
+ // sourceOrStore
+ if (!(sourceOrStore as DataStorage).syncDimensionTypes(dimInfoList)) {
+ dimInfoList = createDimensions(sourceOrStore, zrUtil.extend(createDimensionOptions, {
+ omitUnusedDimensions: true
+ }));
+ // Fallback
+ firstCategoryDimIndex = injectOrdinalMeta(
+ dimInfoList, opt.createInvertedIndices, coordSysInfo
+ );
+ sourceOrStore = source;
+ }
+ }
Review comment:
This code will try to create a dimensions list that only include needed and use `syncDimensionTypes(dimInfoList)` to sync to the storage. If fails to sync, it will recreate a full dimensions list which will be used when recreating the DataStorage. I updated the comment a bit to explain it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675147855
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
> Only in SeriesData will omit the unused dimensions. The cloneAllDimensionInfo used in data transform has no relationship with it. It uses the `dimension define from the source.
Oh yes. Do not need to care `cloneAllDimensionInfo`.
Only the test case above should be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r674500747
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,78 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
+ return upSourceMgrList[0]
+ && upSourceMgrList[0].getSource(sourceIndex);
+ }
+ return source;
+ }
+
+ /**
+ * Will return undefined if source don't have dimensions.
+ *
+ * Only available for series.
+ */
+ getDataStorage(): DataStorage | undefined {
+ if (__DEV__) {
+ assert(isSeries(this._sourceHost), 'Can only call getDataStorage on series source manager.');
+ }
+ const source = this.getSource(0);
+ const dimensionsDefine = source.dimensionsDefine;
+
+ // Source from endpoint(usually series) will be read differently
+ // when seriesLayoutBy or startIndex(which is affected by sourceHeader) are different.
+ // So we use this three props as key.
+ const sourceReadKey = source.seriesLayoutBy
+ + '$$'
+ + source.startIndex
+ + (dimensionsDefine ? map(dimensionsDefine, def => def.name).join('$$') : '');
+
+ return this._innerGetDataStorage(source, sourceReadKey);
+ }
+
+ private _innerGetDataStorage(seriesSource: Source, sourceReadKey: string): DataStorage | undefined {
+ // TODO Can use other sourceIndex?
+ const sourceIndex = 0;
+
+ const storeList = this._storeList;
+
+ let cachedStoreMap = storeList[sourceIndex];
+
+ if (!cachedStoreMap) {
+ cachedStoreMap = storeList[sourceIndex] = {};
+ }
+
+ let cachedStore = cachedStoreMap[sourceReadKey];
+ if (!cachedStore) {
+ const upSourceMgr = this._getUpstreamSourceManagers()[0];
+
+ if (isSeries(this._sourceHost) && upSourceMgr) {
+ cachedStore = upSourceMgr._innerGetDataStorage(seriesSource, sourceReadKey);
+ }
+ else {
+ // Always create datastorage based on source from series.
+ const dimensionsDefine = seriesSource.dimensionsDefine;
+ // Can't create a store if don't know dimension..
+ if (seriesSource && dimensionsDefine) {
Review comment:
In this case. Dataset will still generate `dimensionsDefine` for the source. `dimensionsDefine` is only undefined when the source is `series.data` instead of dataset. In which case sharing storage is meaningless.
Going to add comment to explain it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r682292652
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
@100pah Understand. The rule of picking the last dimension as value is broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r673737621
##########
File path: src/data/helper/sourceManager.ts
##########
@@ -346,8 +350,79 @@ export class SourceManager {
* @param sourceIndex By defualt 0, means "main source".
* Most cases there is only one source.
*/
- getSource(sourceIndex?: number) {
- return this._sourceList[sourceIndex || 0];
+ getSource(sourceIndex?: number): Source {
+ sourceIndex = sourceIndex || 0;
+ const source = this._sourceList[sourceIndex];
+ if (!source) {
+ // Series may share source instance with dataset.
+ const upSourceMgrList = this._getUpstreamSourceManagers();
Review comment:
Oh, get it.
Do we need to add some brief comment or the link of this explantation in source code? Other one may not get the reason of this consideration when reading code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675303987
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
Review comment:
I split the original logic in https://github.com/apache/echarts/blob/master/src/data/helper/completeDimensions.ts#L115 into two:
https://github.com/apache/echarts/blob/3de73abe262912b928e23b6eab6ac568a67bd7d7/src/data/helper/createDimensions.ts#L160
and
https://github.com/apache/echarts/blob/3de73abe262912b928e23b6eab6ac568a67bd7d7/src/data/helper/createDimensions.ts#L139
Because I need to create and set the result dimension only when it's used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] 100pah commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
100pah commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r684813263
##########
File path: src/data/helper/createDimensions.ts
##########
@@ -44,33 +51,350 @@ export type CoordDimensionDefinitionLoose = CoordDimensionDefinition['name'] | C
export type CreateDimensionsParams = {
coordDimensions?: CoordDimensionDefinitionLoose[],
+ /**
+ * Will use `source.dimensionsDefine` if not given.
+ */
dimensionsDefine?: DimensionDefinitionLoose[],
+ /**
+ * Will use `source.encodeDefine` if not given.
+ */
encodeDefine?: HashMap<OptionEncodeValue, DimensionName> | OptionEncode,
dimensionsCount?: number,
+ /**
+ * Make default encode if user not specified.
+ */
encodeDefaulter?: EncodeDefaulter,
generateCoord?: string,
- generateCoordCount?: number
+ generateCoordCount?: number,
+
+ /**
+ * If omit unused dimension
+ * Used to improve the performance on high dimension data.
+ */
+ omitUnusedDimensions?: boolean
};
/**
- * @param opt.coordDimensions
- * @param opt.dimensionsDefine By default `source.dimensionsDefine` Overwrite source define.
- * @param opt.encodeDefine By default `source.encodeDefine` Overwrite source define.
- * @param opt.encodeDefaulter Make default encode if user not specified.
+ * This method builds the relationship between:
+ * + "what the coord sys or series requires (see `coordDimensions`)",
+ * + "what the user defines (in `encode` and `dimensions`, see `opt.dimensionsDefine` and `opt.encodeDefine`)"
+ * + "what the data source provids (see `source`)".
+ *
+ * Some guess strategy will be adapted if user does not define something.
+ * If no 'value' dimension specified, the first no-named dimension will be
+ * named as 'value'.
*/
export default function createDimensions(
// TODO: TYPE completeDimensions type
- source: Source | List | OptionSourceData,
+ source: Source | SeriesData | OptionSourceData | DataStorage,
opt?: CreateDimensionsParams
): DataDimensionInfo[] {
+ if (source instanceof DataStorage) {
+ source = source.getSource();
+ }
+ else if (source instanceof SeriesData) {
+ source = source.getStorage().getSource();
+ }
+ else if (!isSourceInstance(source)) {
+ source = createSourceFromSeriesDataOption(source as OptionSourceData);
+ }
+
opt = opt || {};
- return completeDimensions(opt.coordDimensions || [], source, {
- // FIXME:TS detect whether source then call `.dimensionsDefine` and `.encodeDefine`?
- dimsDef: opt.dimensionsDefine || (source as Source).dimensionsDefine,
- encodeDef: opt.encodeDefine || (source as Source).encodeDefine,
- dimCount: opt.dimensionsCount,
- encodeDefaulter: opt.encodeDefaulter,
- generateCoord: opt.generateCoord,
- generateCoordCount: opt.generateCoordCount
+
+ const sysDims = opt.coordDimensions || [];
+ const dimsDef = opt.dimensionsDefine || source.dimensionsDefine || [];
+ const coordDimNameMap = createHashMap<true, DimensionName>();
+ const result: DataDimensionInfo[] = [];
+ const omitUnusedDimensions = opt.omitUnusedDimensions;
+ const isUsingSourceDimensionsDef = dimsDef === source.dimensionsDefine;
+ // Try to cache the dimNameMap if the dimensionsDefine is from source.
+ const canCacheDimNameMap = (isUsingSourceDimensionsDef && omitUnusedDimensions);
+ let dataDimNameMap = canCacheDimNameMap && inner(source).dimNameMap;
+ let needsUpdateDataDimNameMap = false;
+ if (!dataDimNameMap) {
+ needsUpdateDataDimNameMap = true;
+ dataDimNameMap = createHashMap<DimensionIndex, DimensionName>();
+ }
+
+ const dimCount = getDimCount(source, sysDims, dimsDef, opt.dimensionsCount);
+
+ let encodeDef = opt.encodeDefine;
+ if (!encodeDef && opt.encodeDefaulter) {
+ encodeDef = opt.encodeDefaulter(source, dimCount);
+ }
+ const encodeDefMap = createHashMap<DimensionIndex[] | false, DimensionName>(encodeDef as any);
+
+ const indicesMap = new CtorInt32Array(dimCount);
+ for (let i = 0; i < indicesMap.length; i++) {
+ indicesMap[i] = -1;
+ }
+ function getResultItem(dimIdx: number) {
+ const idx = indicesMap[dimIdx];
+ if (idx < 0) {
+ const dimDefItemRaw = dimsDef[dimIdx];
+ const dimDefItem = isObject(dimDefItemRaw) ? dimDefItemRaw : { name: dimDefItemRaw };
+ const resultItem = new DataDimensionInfo();
+ const userDimName = dimDefItem.name;
+ if (dataDimNameMap.get(userDimName) != null) {
+ resultItem.name = resultItem.displayName = userDimName;
+ }
+ dimDefItem.type != null && (resultItem.type = dimDefItem.type);
+ dimDefItem.displayName != null && (resultItem.displayName = dimDefItem.displayName);
+ const newIdx = result.length;
+ indicesMap[dimIdx] = newIdx;
+ result.push(resultItem);
+ return resultItem;
+ }
+ return result[idx];
+ }
+
+ if (needsUpdateDataDimNameMap) {
+ for (let i = 0; i < dimCount; i++) {
+ const dimDefItemRaw = dimsDef[i];
+ const userDimName = isObject(dimDefItemRaw) ? dimDefItemRaw.name : dimDefItemRaw;
+ // Name will be applied later for avoiding duplication.
+ if (userDimName != null && dataDimNameMap.get(userDimName) == null) {
+ // Only if `series.dimensions` is defined in option
+ // displayName, will be set, and dimension will be diplayed vertically in
+ // tooltip by default.
+ dataDimNameMap.set(userDimName, i);
+ }
+ }
+ if (canCacheDimNameMap) {
+ inner(source).dimNameMap = dataDimNameMap;
+ }
+ }
+ if (!omitUnusedDimensions) {
+ for (let i = 0; i < dimCount; i++) {
+ getResultItem(i);
+ }
+ }
+
+ // Set `coordDim` and `coordDimIndex` by `encodeDefMap` and normalize `encodeDefMap`.
+ encodeDefMap.each(function (dataDimsRaw, coordDim) {
+ const dataDims = normalizeToArray(dataDimsRaw as []).slice();
+
+ // Note: It is allowed that `dataDims.length` is `0`, e.g., options is
+ // `{encode: {x: -1, y: 1}}`. Should not filter anything in
+ // this case.
+ if (dataDims.length === 1 && !isString(dataDims[0]) && dataDims[0] < 0) {
+ encodeDefMap.set(coordDim, false);
+ return;
+ }
+
+ const validDataDims = encodeDefMap.set(coordDim, []) as DimensionIndex[];
+ each(dataDims, function (resultDimIdxOrName, idx) {
+ // The input resultDimIdx can be dim name or index.
+ const resultDimIdx = isString(resultDimIdxOrName)
+ ? dataDimNameMap.get(resultDimIdxOrName)
+ : resultDimIdxOrName;
+ if (resultDimIdx != null && resultDimIdx < dimCount) {
+ validDataDims[idx] = resultDimIdx;
+ applyDim(getResultItem(resultDimIdx), coordDim, idx);
+ }
+ });
+ });
+
+ // Apply templetes and default order from `sysDims`.
+ let availDimIdx = 0;
+ each(sysDims, function (sysDimItemRaw) {
+ let coordDim: DimensionName;
+ let sysDimItemDimsDef: CoordDimensionDefinition['dimsDef'];
+ let sysDimItemOtherDims: CoordDimensionDefinition['otherDims'];
+ let sysDimItem: CoordDimensionDefinition;
+ if (isString(sysDimItemRaw)) {
+ coordDim = sysDimItemRaw;
+ sysDimItem = {} as CoordDimensionDefinition;
+ }
+ else {
+ sysDimItem = sysDimItemRaw;
+ coordDim = sysDimItem.name;
+ const ordinalMeta = sysDimItem.ordinalMeta;
+ sysDimItem.ordinalMeta = null;
+ sysDimItem = extend({}, sysDimItem);
+ sysDimItem.ordinalMeta = ordinalMeta;
+ // `coordDimIndex` should not be set directly.
+ sysDimItemDimsDef = sysDimItem.dimsDef;
+ sysDimItemOtherDims = sysDimItem.otherDims;
+ sysDimItem.name = sysDimItem.coordDim = sysDimItem.coordDimIndex =
+ sysDimItem.dimsDef = sysDimItem.otherDims = null;
+ }
+
+ let dataDims = encodeDefMap.get(coordDim);
+
+ // negative resultDimIdx means no need to mapping.
+ if (dataDims === false) {
+ return;
+ }
+
+ dataDims = normalizeToArray(dataDims);
+
+ // dimensions provides default dim sequences.
+ if (!dataDims.length) {
+ for (let i = 0; i < (sysDimItemDimsDef && sysDimItemDimsDef.length || 1); i++) {
+ while (availDimIdx < dimCount && getResultItem(availDimIdx).coordDim != null) {
+ availDimIdx++;
+ }
+ availDimIdx < dimCount && dataDims.push(availDimIdx++);
+ }
+ }
+
+ // Apply templates.
+ each(dataDims, function (resultDimIdx, coordDimIndex) {
+ const resultItem = getResultItem(resultDimIdx);
+ // Coordinate system has a higher priority on dim type than source.
+ if (isUsingSourceDimensionsDef && sysDimItem.type != null) {
+ resultItem.type = sysDimItem.type;
+ }
+ applyDim(defaults(resultItem, sysDimItem), coordDim, coordDimIndex);
+ if (resultItem.name == null && sysDimItemDimsDef) {
+ let sysDimItemDimsDefItem = sysDimItemDimsDef[coordDimIndex];
+ !isObject(sysDimItemDimsDefItem) && (sysDimItemDimsDefItem = {name: sysDimItemDimsDefItem});
+ resultItem.name = resultItem.displayName = sysDimItemDimsDefItem.name;
+ resultItem.defaultTooltip = sysDimItemDimsDefItem.defaultTooltip;
+ }
+ // FIXME refactor, currently only used in case: {otherDims: {tooltip: false}}
+ sysDimItemOtherDims && defaults(resultItem.otherDims, sysDimItemOtherDims);
+ });
});
+
+ function applyDim(resultItem: DataDimensionInfo, coordDim: DimensionName, coordDimIndex: DimensionIndex) {
+ if (VISUAL_DIMENSIONS.get(coordDim as keyof DataVisualDimensions) != null) {
+ resultItem.otherDims[coordDim as keyof DataVisualDimensions] = coordDimIndex;
+ }
+ else {
+ resultItem.coordDim = coordDim;
+ resultItem.coordDimIndex = coordDimIndex;
+ coordDimNameMap.set(coordDim, true);
+ }
+ }
+
+ // Make sure the first extra dim is 'value'.
+ const generateCoord = opt.generateCoord;
+ let generateCoordCount = opt.generateCoordCount;
+ const fromZero = generateCoordCount != null;
+ generateCoordCount = generateCoord ? (generateCoordCount || 1) : 0;
+ const extra = generateCoord || 'value';
+ let coordDimNameAutoIdx = 0;
+ let dataDimNameAutoIdx = 0;
+
+ // Set dim `name` and other `coordDim` and other props.
+ if (!omitUnusedDimensions) {
+ for (let resultDimIdx = 0; resultDimIdx < dimCount; resultDimIdx++) {
+ const resultItem = getResultItem(resultDimIdx);
+ const coordDim = resultItem.coordDim;
+
+ if (coordDim == null) {
+ const res = genName(
+ extra, coordDimNameMap, coordDimNameAutoIdx, fromZero
+ );
+ coordDimNameAutoIdx = res.autoIdx;
+ resultItem.coordDim = res.name;
+ resultItem.coordDimIndex = 0;
+ // Series specified generateCoord is using out.
+ if (!generateCoord || generateCoordCount <= 0) {
+ resultItem.isExtraCoord = true;
+ }
+ generateCoordCount--;
+ }
+
+ if (resultItem.name == null) {
+ const res = genName(
+ resultItem.coordDim, dataDimNameMap, dataDimNameAutoIdx, false
+ );
+ resultItem.name = res.name;
+ dataDimNameAutoIdx = res.autoIdx;
+ }
+
+ if (resultItem.type == null
+ && (
+ guessOrdinal(source, resultDimIdx) === BE_ORDINAL.Must
+ // Consider the case:
+ // {
+ // dataset: {source: [
+ // ['2001', 123],
+ // ['2002', 456],
+ // ...
+ // ['The others', 987],
+ // ]},
+ // series: {type: 'pie'}
+ // }
+ // The first colum should better be treated as a "ordinal" although it
+ // might not able to be detected as an "ordinal" by `guessOrdinal`.
+ || (resultItem.isExtraCoord
+ && (resultItem.otherDims.itemName != null
+ || resultItem.otherDims.seriesName != null
+ )
+ )
+ )
+ ) {
+ resultItem.type = 'ordinal';
+ }
+ }
+ return result;
+ }
+ else {
+ // Sort dimensions
+ const toSort = [];
+ for (let i = 0; i < indicesMap.length; i++) {
+ if (indicesMap[i] >= 0) {
+ toSort.push({ i, o: result[indicesMap[i]]});
+ }
+ }
+ toSort.sort((a, b) => a.i - b.i);
Review comment:
The index of `SeriesData['dimensions']` and `SeriesData['getDimension']` is not the dimensionIndex that users known, but the internal dimension index if dimension count > 30.
But there are someplace output that index to users:
+ in `CustomView`: `wrapEncodeDef(customSeries.getData())`),
or use user input dimension index on `SeriesData['dimensions']`/`SeriesData['getDimension']`:
+ in `Customview`: `value()`
And `RadarView` has bug if use `encode` before this PR:
```js
encode: {
value: [1, 2, 3, 4, 5, 6],
itemName: 7
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org
[GitHub] [echarts] pissang commented on a change in pull request #15355: dataset: fix dataset performance drops signifcantly on high dimensions data.
Posted by GitBox <gi...@apache.org>.
pissang commented on a change in pull request #15355:
URL: https://github.com/apache/echarts/pull/15355#discussion_r675295018
##########
File path: src/data/DataStorage.ts
##########
@@ -0,0 +1,1281 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
+import { assert, clone, isFunction, keys, map, reduce } from 'zrender/src/core/util';
+import {
+ Dictionary,
+ DimensionIndex,
+ DimensionName,
+ OptionDataItem,
+ ParsedValue,
+ ParsedValueNumeric
+} from '../util/types';
+import { DataProvider } from './helper/dataProvider';
+import { parseDataValue } from './helper/dataValueHelper';
+import OrdinalMeta from './OrdinalMeta';
+import { Source } from './Source';
+
+const UNDEFINED = 'undefined';
+/* global Float64Array, Int32Array, Uint32Array, Uint16Array */
+
+// Caution: MUST not use `new CtorUint32Array(arr, 0, len)`, because the Ctor of array is
+// different from the Ctor of typed array.
+export const CtorUint32Array = typeof Uint32Array === UNDEFINED ? Array : Uint32Array;
+export const CtorUint16Array = typeof Uint16Array === UNDEFINED ? Array : Uint16Array;
+export const CtorInt32Array = typeof Int32Array === UNDEFINED ? Array : Int32Array;
+export const CtorFloat64Array = typeof Float64Array === UNDEFINED ? Array : Float64Array;
+/**
+ * Multi dimensional data storage
+ */
+const dataCtors = {
+ 'float': CtorFloat64Array,
+ 'int': CtorInt32Array,
+ // Ordinal data type can be string or int
+ 'ordinal': Array,
+ 'number': Array,
+ 'time': CtorFloat64Array
+} as const;
+
+// Dim with same category can be convert between.
+const dataCtorCategory = {
+ 'float': 0, 'time': 0,
+ 'number': 1, 'int': 2,
+ 'ordinal': 3
+};
+
+export type DataStorageDimensionType = keyof typeof dataCtors;
+
+type DataTypedArray = Uint32Array | Int32Array | Uint16Array | Float64Array;
+type DataTypedArrayConstructor = typeof Uint32Array | typeof Int32Array | typeof Uint16Array | typeof Float64Array;
+type DataArrayLikeConstructor = typeof Array | DataTypedArrayConstructor;
+
+
+type DataValueChunk = ArrayLike<ParsedValue>;
+
+// If Ctx not specified, use List as Ctx
+type EachCb0 = (idx: number) => void;
+type EachCb1 = (x: ParsedValue, idx: number) => void;
+type EachCb2 = (x: ParsedValue, y: ParsedValue, idx: number) => void;
+type EachCb = (...args: any) => void;
+type FilterCb0 = (idx: number) => boolean;
+type FilterCb1 = (x: ParsedValue, idx: number) => boolean;
+type FilterCb = (...args: any) => boolean;
+type MapArrayCb = (...args: any) => any;
+type MapCb = (...args: any) => ParsedValue | ParsedValue[];
+
+export type DimValueGetter = (
+ this: DataStorage,
+ dataItem: any,
+ dimName: string,
+ dataIndex: number,
+ dimIndex: DimensionIndex
+) => ParsedValue;
+
+export interface DataStorageDimensionInfo {
+ type?: DataStorageDimensionType; // Default to be float.
+ name?: string;
+ /**
+ * When using category axis.
+ * Category strings will be collected and stored in ordinalMeta.categories.
+ * And storage will store the index of categories.
+ */
+ ordinalMeta?: OrdinalMeta,
+
+ /**
+ * Offset for ordinal parsing and collect
+ */
+ ordinalOffset?: number
+}
+
+let defaultDimValueGetters: {[sourceFormat: string]: DimValueGetter};
+
+function getIndicesCtor(rawCount: number): DataArrayLikeConstructor {
+ // The possible max value in this._indicies is always this._rawCount despite of filtering.
+ return rawCount > 65535 ? CtorUint32Array : CtorUint16Array;
+};
+function getInitialExtent(): [number, number] {
+ return [Infinity, -Infinity];
+};
+function cloneChunk(originalChunk: DataValueChunk): DataValueChunk {
+ const Ctor = originalChunk.constructor;
+ // Only shallow clone is enough when Array.
+ return Ctor === Array
+ ? (originalChunk as Array<ParsedValue>).slice()
+ : new (Ctor as DataTypedArrayConstructor)(originalChunk as DataTypedArray);
+}
+
+function prepareStorage(
+ storage: DataValueChunk[],
+ dimIdx: number,
+ dimType: DataStorageDimensionType,
+ end: number,
+ append?: boolean
+): void {
+ const DataCtor = dataCtors[dimType || 'float'];
+
+ if (append) {
+ const oldStore = storage[dimIdx];
+ const oldLen = oldStore && oldStore.length;
+ if (!(oldLen === end)) {
+ const newStore = new DataCtor(end);
+ // The cost of the copy is probably inconsiderable
+ // within the initial chunkSize.
+ for (let j = 0; j < oldLen; j++) {
+ newStore[j] = oldStore[j];
+ }
+ storage[dimIdx] = newStore;
+ }
+ }
+ else {
+ storage[dimIdx] = new DataCtor(end);
+ }
+};
+class DataStorage {
+ private _chunks: DataValueChunk[] = [];
+
+ private _provider: DataProvider;
+
+ // Raw extent will not be cloned, but only transfered.
+ // It will not be calculated util needed.
+ private _rawExtent: [number, number][] = [];
+
+ private _extent: [number, number][] = [];
+
+ // Indices stores the indices of data subset after filtered.
+ // This data subset will be used in chart.
+ private _indices: ArrayLike<any>;
+
+ private _count: number = 0;
+ private _rawCount: number = 0;
+
+ private _dimensions: DataStorageDimensionInfo[];
+ private _dimValueGetter: DimValueGetter;
+
+ defaultDimValueGetter: DimValueGetter;
+ /**
+ * Initialize from data
+ * @param data source or data or data provider.
+ */
+ initData(
+ provider: DataProvider,
+ dimensions: DataStorageDimensionInfo[],
+ dimValueGetter?: DimValueGetter
+ ): void {
+ if (__DEV__) {
+ assert(
+ isFunction(provider.getItem) && isFunction(provider.count),
+ 'Inavlid data provider.'
+ );
+ }
+
+ this._provider = provider;
+
+ // Clear
+ this._chunks = [];
+ this._indices = null;
+ this.getRawIndex = this._getRawIdxIdentity;
+
+ const defaultGetter = this.defaultDimValueGetter =
+ defaultDimValueGetters[provider.getSource().sourceFormat];
+ // Default dim value getter
+ this._dimValueGetter = dimValueGetter || defaultGetter;
+
+ const emptyObj = {};
+
+ // Reset raw extent.
+ this._rawExtent = [];
+ this._dimensions = map(dimensions, dim => ({
+ // Only pick these two props. Not leak other properties like orderMeta.
+ type: dim.type,
+ name: dim.name
+ }) as DataStorageDimensionInfo);
+
+ const dimensionsIdxMap: Dictionary<number> = {};
+ let prefix = '';
+
+ // Needs to add prefix if key is used in object prototype
+ for (let i = 0; i < dimensions.length; i++) {
+ const name = dimensions[i].name;
+ if ((emptyObj as any)[name] != null) {
+ prefix = '$$';
+ }
+ }
+ for (let i = 0; i < dimensions.length; i++) {
+ const dim = dimensions[i];
+ const name = dim.name;
+ dimensionsIdxMap[prefix + name] = i;
+ }
+
+ // We use different functions because it may be a hotspot code.
+ this.getDimensionIndex = prefix ? function (dim) {
+ return dimensionsIdxMap[prefix + dim];
+ } : function (dim) {
+ return dimensionsIdxMap[dim];
+ };
+
+ this._initDataFromProvider(0, provider.count());
+ }
+
+ getProvider(): DataProvider {
+ return this._provider;
+ }
+
+ getSource(): Source {
+ return this._provider.getSource();
+ }
+
+ getDimensionIndex: (dim: DimensionName) => number;
+
+ getDimensionCount() {
+ return this._dimensions.length;
+ }
+
+ collectOrdinalMeta(
+ dimIdx: number,
+ ordinalMeta: OrdinalMeta
+ ) {
+ const chunk = this._chunks[dimIdx];
+ const dim = this._dimensions[dimIdx];
+ const rawExtents = this._rawExtent;
+
+ const offset = dim.ordinalOffset || 0;
+ const len = chunk.length;
+
+ if (offset === 0) {
+ // We need to reset the rawExtent if collect is from start.
+ // Because this dimension may be guessed as number and calcuating a wrong extent.
+ rawExtents[dimIdx] = getInitialExtent();
+ }
+
+ const dimRawExtent = rawExtents[dimIdx];
+
+ // Parse from previous data offset. len may be changed after appendData
+ for (let i = offset; i < len; i++) {
+ const val = (chunk as any)[i] = ordinalMeta.parseAndCollect(chunk[i]);
+ dimRawExtent[0] = Math.min(val, dimRawExtent[0]);
+ dimRawExtent[1] = Math.max(val, dimRawExtent[1]);
+ }
+
+ dim.ordinalMeta = ordinalMeta;
+ dim.ordinalOffset = len;
+ dim.type = 'ordinal'; // Force to be ordinal
+ }
+
+ getOrdinalMeta(dimIdx: number) {
+ const dimInfo = this._dimensions[dimIdx];
+ const ordinalMeta = dimInfo.ordinalMeta;
+ return ordinalMeta;
+ }
+
+ /**
+ * If we using dataset.
+ * Dimensions types may only know when we initializing series.
+ * So we need to sync the type back to storage when initlializing SeriesData back
+ *
+ * Will return false if dimension type has been known and different from given.
+ * We need to recreate a new data storage in this case.
+ */
+ // TODO Can't sure what's frequency will this validate fail and cause datastorage recreate.
Review comment:
I've commented about some changes. Now ordinalMeta should be in the key for query and caching OrdinalMeta. No more validation and recreation should happen when ordinaMeta is not the same.(Still there is a `canUse` check to avoid some unexpected scenarios but it should not exist in our cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@echarts.apache.org
For additional commands, e-mail: commits-help@echarts.apache.org